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PREFACE 

This  report  deals  with  a  research  project  between  the  sponsoring 
activity,  the  Personnel  Research  Division  of  the  Bureau  of  Naval  Personnel, 
and  the  Naval  Postgraduate  School.   The  topics  examined  in  this  report  are: 
(1)   Current  procedures  and  recommended,  improved  procedures  for  evaluating 
the  performance  and  effectiveness  of  the  Navy  Personnel  Research  and  Devel- 
opment Laboratory  and  the  Navy  Personnel  and  Training  Research  Laboratory 
and  (2)  administrative  and  managerial  practices  for  encouraging  greater 
application  of  new  research  techniques  and  procedures  in  the  aforementioned 
laboratories.   Subsequent  reports  will  deal  with  the  further  development 
and  application  of  evaluative  procedures,  effectiveness  measures,  and  organ- 
izational principles  for  personnel  and  training  research. 

The  study  was  performed  under  the  auspices  of  Capt.  G.  F.  Britner, 
Division  Director,  and  Mr.  A.  A.  Sjoholm,  Technical  Director,  Personnel 
Research  Division. 

The  author  thanks  Dr.  E.  I.  Jones  and  Mr.  E.  M.  Ramras ,  technical 
directors  of  the  Navy  Personnel  and  Training  Research  Laboratory  and  Navy 
Personnel  Research  and  Development  Laboratory,  respectively,  for  reviewing 
the  draft  of  this  study  and  making  appropriate  suggestions  for  the  final 
product  and  future  direction  of  the  research  effort. 
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I .   INTRODUCTION 

The  two  problems  analyzed  in  t-his  report  are  methods  for  evalu- 
ating the  performance  effectiveness  of  the  Navy's  in-house  personnel 
research  laboratories  and  means  for  improving  their  performance  through 
the  adoption  and  application  of  new  techniques  and  procedures.   These 
two  problems  can  be  considered  together  since  they  are  related  to  the 
more  general  question,  What  characterizes  an  effective,  in-house,  Federal 
laboratory?   They  are  related  to  this  question  because,  first,  the 
characteristics  of  an  effective  laboratory  could  provide  the  basis  for 
devising  criteria  to  evaluate  the  effectiveness  of  a  given  laboratory. 
And  such  characteristics,  when  known,  might  also  provide  guidance  for 
recommending  managerial  and  administrative  practices  to  enhance  the 
effectiveness  of  a  particular  laboratory  through  the  use  of  new  and 
innovative  procedures. 

Accordingly,  the  approach  taken  in  this  report  is  to  begin  with 
the  identification  of  salient  features  of  organizational  climates  that 
appear  to  be  important  to  effective  performance  within  research  and 
development  (R&D)  laboratories.   Next,  the  environment  or  climate  of 
Federal,  in-house  laboratories  will  be  examined.   Following  this,  attempts 
to  assess  and  evaluate  the  effectiveness  of  R&D  organization  will  be 
examined  with  particular  reference  to  in-house  laboratories.   Finally, 
the  specific  procedures  currently  in  use  to  evaluate  the  effectiveness 
of  the  Navy's  in-house,  personnel  research  activities  will  be  reviewed, 
and  recommendations  will  be  made  for  improved  procedures  based  on  the 
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information  developed  in  the  preceding  investigation. 

Recommendations  for  administrative  and  managerial  practices  to 
foster  the  adoption  and  application  of  new  research  techniques  and  pro- 
cedures in  the  Navy's  personnel  research  laboratories  will  arise  as  a 
by-product  of  the  attempt  to  identify  evaluation  criteria  and  methods 
for  assessing  the  effectiveness  of  laboratories  in  the  manner  contem- 
plated. 


-3- 


II.   CHARACTERISTICS  OF  RESEARCH  AND  DEVELOPMENT  LABORATORIES 

A.   GENERAL  TRENDS 

Leaving  undefined  for  the  present  what  constitutes  effectiveness 
in  R&D  activities,  it  seems  appropriate  to  assume  that  managerial  and 
organizational  practices  that  are  conducive  to  effective  operations  in 
any  type  of  organization  are  also  applicable  to  R&D  organizations,  such 
as  laboratories.   This  assumption  is  examined  in  detail  by  Argyris  (1968) 
in  its  negative  aspects.   He  explains  how  the  detrimental  features  of  a 
typical  pyramidal  organizational  structure  can  be  magnified  in  the  dy- 
namic life  of  an  R&D  organization.   The  thrust  of  his  argument  is  that 
typical  pyramidal  organizations,  when  imposed  on  an  R&D  activity,  may  be 
conducive  to  the  steady  deterioration  of  the  activity.   The  deterioration 
is  primarily  the  result  of  improper  controls  imposed  by  the  upper  levels 
of  the  organization  that  are  based  on  mistrust,  antagonism,  and  defen- 
siveness.   As  a  result,  the  system  becomes  increasingly  rigid;  less 
competent  personnel  reach  upper  levels  of  the  organization;  accurate  in- 
formation ceases  to  flow  through  the  system;  paternalism  begins  to 
flourish;  more  time  is  spent  on  generating  valid,  perhaps,  but  not  very 
important  information;  and  nonresearch  activities  —  "selling,"  budgets, 
hiding  bad  news  from  the  top  —  begin  to  take  precedence  over  research 
activities.   The  increasing  frustration  of  the  scientists  in  the  organi- 
zation is  reflected  by  hesitating  to  take  on  responsibility  and  making 
tangential  demands  for  space,  equipment,  assistants,  travel  to  professional 
meetings,  and  the  like.   Good  scientists  leave  the  organization.   As  the 
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R&D  organization  that  is  locked  in  to  this  deteriorating  process  grows 

older,  top  management  becomes  disenchanted  with  it.   Increasing  .  ,  . 

intervention  by  the  line  departments,  tighter  research 
budgets,  more  systematic  evaluation  of  research,  more 
rigorous  (hopefully)  quantitative  indices  for  evaluat- 
ing research  payoff,  more  use  of  charts  to  control  the 
flow  of  research,  closer  link  with  the  marketplace  and 
a  weaker  tie  with  basic  research,  and  finally,  an  in- 
crease in  the  use  of  top-level  committees  to  oversee 
research  can  be  expected.  (Argyris,  1968,  p.  353.) 

The  positive  side  of  this  argument  is  to  design  the  controls, 
organizational  structures,  and  leadership  styles  so  that  the  participants 
".  .  .  may  give  of  themselves  rather  than  give  up_  themselves"  (Argyris, 
1968,  p.  347).   The  basis  for  designing  such  controls  rests  in  openness, 
respect  for  individuality,  trust,  concern,  and  a  willingness  to  permit 
risk-taking  and  innovation. 

Controls,  of  course,  imply  pressure  regardless  of  whether  the 
controls  are  good  or  bad,  positive  or  negative.   To  most  persons,  pressure 
in  a  job  is  an  indication  of  undesirable  stresses.   In  fact,  the  often 
asked  question  about  an  individual  is  his  ability  to  stand  up  under 
pressure.   Thus,  a  logical  next  step,  following  Argyris'  critical  ap- 
praisal of  controls  in  R&D  organizations,  is  an  examination  of  the  role 
of  pressure.   Should  pressure  be  entirely  undesirable?   If  so,  how  are 
controls  to  be  exercised?  Merely  through  reward  systems?  The  question, 
according  to  Hall  and  Lawler  (1971) ,  is  not  whether  pressure  is  good  or 
bad  but  to  determine  when  pressure  is  helpful  and  when  it  is  dysfunc- 
tional.  They  state  that  healthy  persons  have  a  need  to  experience  in- 
ternal pressure  and  will  create  it  if  the  environment  fails  to  provide  it. 
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In  order  to  identify  healthy  pressures  and  their  beneficial  uses, 
it  is  necessary  to  distinguish  between  pressure  and  such  terms  as  conflict, 
stress,  tension,  anxiety,  and  strain.   To  Hall  and  Lawler,  job  pressures 
are  forces  a  person  experiences  which  motivate  him  to  behave  in  partic- 
ular ways  on  the  job,  and  pressure  is  the  experience  of  a  particular 
force.   Strain,  conflict,  and  similar  conceptions  represent  the  oppo- 
sition of  two  or  more  pressures.   With  pressure  thus  defined,  their  re- 
search identified  three  kinds  of  pressure  that  were  mentioned  from 
laboratory  to  laboratory  with  considerable  agreement  as  to  their  impor- 
tance to  research  personnel.   These  were  time,  quality,  and  financial 
responsibility.   Time  was  a  felt  need  to  meet  schedules  and  deadlines, 
and  it  was  typically  externally  imposed.   Quality  pressure  was  a  concern 
for  doing  a  good  technical  job  and  seemed  to  be  generated  by  the  re- 
searchers themselves,  along  with  their  colleagues.   And  finally,  financial 
responsibility  was  felt  as  a  concern  for  the  financial  goal-attainment 
of  the  organization  as  a  whole. 

These  pressures  were  then  related  by  Hall  and  Lawler  (1971)  to 
the  satisfaction  and  job  involvement  of  the  research  worker  and  to  the 
effectiveness  of  the  organization.   Quality  pressure  was  related  posi- 
tively to  both  job  involvement  of  the  individual  professionals  and  to 
the  technical  effectiveness  of  their  laboratories.   Job  involvement  was 
also  related  to  laboratory  effectiveness.   Financial  responsibility 
pressure  was  significantly  correlated  with  both  laboratory  effectiveness 
and  the  satisfaction  of  the  researcher's  need  for  autonomy.   The  impli- 
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cation  here  was  that  the  more  a  person  helps  to  support  himself  and  his 

organization  financially,  the  less  dependent  he  is  on  the  organization. 

Time  pressure,  while  not  negatively  related  to  any  of  the  criteria,  was 

not  related  in  a  systematic  way  to  researcher  needs  or  to  laboratory 

effectiveness. 

Hall  and  Lawler  (1971)  conclude: 

.  .  .  the  most  effective  organizations  are  those  in  which 
the  professional  value  of  technical  excellence  and/or  the 
organizational  value  of  financial  responsibility  are  in- 
ternalized by  the  system  as  a  whole.   These  two  alternatives 
indicate  two  separate  types  of  adaptations  —  pressure  for 
quality  reflects  organizational  adaption  to  professional 
norms,  while  financial  pressure  suggests  the  professionals' 
adaption  to  organizational  values.   (P.  70.) 

This  means,  therefore,  that  in  the  effective  system, 
each  member  feels  "on  top  of"  the  organization;  he  is 
aware  of  the  total  system  as  it  relates  to  its  environment 
and  knows  how  his  job  ties  in  with  the  total  goals  of  the 
organization.   This  self  and  system  awareness  represents 
a  strong  sense  of  organizational  identification  for  the 
person,  and  this  experience  when  widely  shared  is  an  im- 
portant dimension  of  organizational  health.   (P.  71.) 

Pelz  and  Andrews  (1966) ,  in  their  work  on  scientists  in  organi- 
zations, also  conclude  that  "...  the  scientists  and  engineers  whom  we 
studied  did  effective  work  under  conditions  that  were  not  completely 
comfortable,  but  contained  'creative  tensions'  among  forces  pulling  in 
different  directions"  (p.  7).   They  (Pelz  &  Andrews,  1966;  p.  7) 
characterized  broad  features  of  the  environment  of  the  most  productive 
scientists  and  engineers,  as  follows: 

'Effective  scientists  were  self-directed  by  their  own 
ideas,  and  valued  freedom.   But  at  the  same  time  they 
allowed  several  other  people  a  voice  in  shaping  their 
directions;  they  interacted  vigorously  with  colleagues. 
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'Effective  scientists  did  not  limit  their  activities 
either  to  the  world  of  "application"  or  to  the  world 
of  "pure  science"  but  maintained  an  interest  in  both; 
their  work  was  diversified. 

'Effective  scientists  were  not  fully  in  agreement  with 
their  organization  in  terms  of  their  interests;  what 
they  personally  enjoyed  did  not  necessarily  help  them 
advance  in  the  structure. 

'Effective  scientists  tended  to  be  motivated  by  the  same 
kinds  of  things  as  their  colleagues.   At  the  same  time, 
however,  they  differed  from  their  colleagues  in  the 
styles  and  strategies  with  which  they  approached  their 
work. 

"In  effective  older  groups,  the  members  interacted  vig- 
orously and  preferred  each  other  as  collaborators,  yet 
they  held  each  other  at  an  emotional  distance  and  felt 
free  to  disagree  on  technical  strategies. 

From  the  literature  reviewed  above,  it  can  be  said — in  general — 
that  productive  and  efficient  R&D  laboratories  use  organizational  con- 
trols that  embody  pressures  which  are  not  dysfunctional,  and  their  individual 
scientists,  while  preserving  their  own  independence  and  interests,  are 
flexible  and  dedicated  to  both  the  pursuit  of  quality  in  their  work  and 
to  the  welfare  of  their  organizations. 

B.   TRENDS  IN  FEDERAL  LABORATORIES 

Glass  (1970,  p.  11)  writes  that  "Probably  no  class  of  institution 
has  been  studied  and  analyzed,  praised  and  criticized,  organized  and 
reorganized  to  the  degree  that  has  been  the  lot  of  the  Defense  labora- 
tories." His  report  (Glass,  1970)  provides  a  comprehensive  review  of 
the  study,  analysis,  and  evolution  of  Defense  in-house  laboratories. 
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The  status  and  potential  of  Federal  laboratories  are  also  discussed  in 
detail  in  the  report  of  the  hearings  in  March  and  April  1968  of  the  House 
Subcommittee  on  Science,  Research,  and  Development,  frequently  referred 
to  as  the  Daddario  Committee.   (Hereafter  it  will  be  referred  to  as  the 
Subcommittee,  1968,)   Pelz  and  Andrews  (1966)  also  devote  much  of  their 
book  to  scientists  in  government  laboratories.   More  recently,  the  Defense 
Blue  Ribbon  Panel  completed  its  investigation  of  Defense  R&D  activities 
with  recommendations  for  major  changes  and  improvements  in  the  system. 
Especially  germane  to  this  report  is  the  investigation  of  the  Manpower 
Research  Task  Force  of  the  Defense  Science  Board,  which  is  nearing 
completion.   Accordingly,  the  intent  here  is  not  to  retrace  this  path  of 
study  and  analysis,  praise  and  criticism.   Rather,  the  purpose  of  this 
section  is  to  highlight  salient  characteristics  of  Federal  and  especially 
Department  of  Defense  (DoD)  laboratories  that  may  have  a  bearing  on  the 
questions  being  examined  in  this  report. 

If  professionals  in  good  R&D  laboratories  identify  with  their 
organizations  as  mentioned  above,  surely  those  in  Federal  laboratories 
must  do  so  with  respect  to  the  mission  and  purpose  of  their  laboratories 
and  the  problems  that  are  selected  for  investigation.   The  Committee  on 
Federal  Laboratories,  for  example,  found  that  the  single  most  important 
factor  in  laboratory  morale  was  a  sense  of  purpose  on  the  part  of  each 
scientist  and  a  sense  that  the  results  mattered  to  someone  (Subcommittee, 
1968).   Harold  B.  Finger,  Associate  Administrator  for  Organization  and 
Management,  National  Aeronautics  and  Space  Administration  (NASA),  testi- 
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fying  before  the  Subcommittee,  stated  that  the  NASA  experience  confirmed 
the  value  and  importance  of  the  purpose  of  an  organization.   He  noted 
that  it  would  be  very  hard  for  a  research  organization  to  succeed  if  its 
basic  purposes  and  motivation  seemed  unimportant;  on  the  other  hand,  a 
research  organization  which  has  an  important  and  significant  role  to  ful- 
fill has  a  very  good  start  in  the  critical  process  of  building  and  holding 
a  research  competence.   Dr.  Donald  M.  McArthur,  representing  the  Office  of 
the  Director  of  Defense  Research  and  Engineering  (DDR&E)  before  the  Sub- 
committee, stated  that  the  quality  of  a  laboratory  ".  .  .  really  comes 
down  in  the  end  to  a  number  of  criteria,  but  an  overriding  criteria  is 
mission.   Does  it  have  a  sense  of  purpose?   Does  it  have  high-quality 
people,  and  are  they  performing  well?"   (Subcommittee,  1968;  p.  151). 

With  respect  to  the  mission  or  purpose  of  Federal  laboratories, 
Dr.  Donald  F.  Hornig,  director  of  the  Office  of  Science  and  Technology, 
Executive  Office  of  the  President,  spoke  to  the  Subcommittee  of  some  things 
that  are  not.   He  emphasized  the  point  that  these  laboratories  do  not  exist 
to  "accumulate  a  pile  of  knowledge,"  that  they  do  not  "turn  over  the  grains 
of  sand  in  the  Sahara  Desert  and  examine  them  one  by  one,"  that  theirs  is 
not  the  purpose  to  do  research  for  its  own  sake.   Their  task  is  to  solve 
present  problems  and  lay  the  foundation  for  the  future;  they  must  produce 
the  ideas  on  which  the  next  generation  of  their  parent  agencies'  activities 
will  be  based.   Accordingly,  the  laboratories  need  to  have  meaningful  prob- 
lems to  work  on,  where  the  end  results  of  what  they  do  will  be  visible  and 
on  which  they  could  be  judged.   He  mentioned  a  "multiplier  effect"  of  the 
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product  of  research  activities.   That  is,  their  effect  is  on  present  and 
future  programs  of  the  entire  Nation  usually  involving  expenditures  many 
times  greater  than  those  directly  involved  in  the  laboratory  operations. 
Because  of  the  "multiplier  effect,"  Dr.  Hornig  stated  that  the  utility  of 
Federal  laboratories  must  be  viewed  in  the  context  of  their  overall  contri- 
bution to  national  progress  rather  than  in  the  narrow  context  of  the  admin- 
istration of  the  laboratories  themselves. 

The  mission  or  purpose  of  any  government  laboratory  is  operation- 
ally manifested  in  the  problems  that  are  selected  for  investigation  and 
solution   The  speakers  before  the  Subcommittee  were  in  close  agreement 
over  the  fact  that  problem  selection  was  the  critical  and  most  difficult 
task  facing  laboratories  and  especially  the  laboratory  director.   They  noted 
that  the  input  for  making  the  decisions  needed  to  come  up  from  "the  bench" 
and  the  laboratorv  management  as  well  as  from  echelons  above  the  laboratory. 
Hornig  stated  that  it  is  not  usually  true  that  a  person  sitting  at  the  top 
of  the  pyramid   like  himself,  could  sav  what  the  problems  really  are  and 
what  real  technical  problems  are  "soluble  next  year".   He  continued:   "... 
whether  you  make  progress  depends  on  picking  the  right  detailed  problems, 
those  which  are  ripe  for  solving  at  a  given  time.   That  requires  keen  tech- 
nical insight."   (Subcommittee,  1968;  p.  y.) 

Problem  selection  with  respect  to  the  Navy's  personnel  research  lab- 
oratories is  an  extremely  complex  procedure  that  requires  interaction  between 
the  highest  echelons  of  Navy  management,  several  levels  of  middle  management, 
and  the  outlook  at  the  "bench"  in  the  laboratories  themselves.   Problem 
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selection  depends,  first,  on  establishing  goals  and  objectives  that  define 
the  total  R&D  program  for  the  laboratories.   This  is  accomplished,  in  turn, 
by  considering  guidance  from  program  sponsors  and  planning  documents,  spe- 
cific consumer  requests,  policy  changes,  and  the  results  of  task-force  in- 
vestigations (such  as  the  recent  Blue  Ribbon  Defense  Panel  recommendations). 
On  the  other  hand,  consideration  in  program  definition  must  be  given  to  the 
current  ongoing  program,  the  timing  of  requirements  and  expected  completion 
of  products,  the  capabilities  of  the  laboratories,  their  stated  missions, 
and  overall  funding  constraints.   Finally,  room  must  be  provided  for  self- 
generated  ideas. 

Once  the  goals  and  objectives  of  the  overall  program  have  been  de- 
fined, problem  selection  will  then  depend  on  priority  determination  and 
an  allocation  of  the  necessary  resources.   Again,  on  one  hand,  the  poten- 
tial impact,  significance,  and  payoff  of  the  solution  of  a  problem  are 
prime  considerations,  along  with  requests  from  program  sponsors  and  con- 
sumers.  The  current  state  of  the  art,  requirements  for  additional  research, 
and  the  probability  of  success  are  also  important  factors.   Again,  there 
are,  on  the  other  hand,  the  capabilities,  specialization,  and  performance 
history  of  those  who  would  work  on  a  problem  and  the  investment  required  in 
facilities,  capital  equipment,  and  personnel.   Finally,  there  is  a  broad 
allocation  problem  of  the  division  in  resource  committment  between  research 
and  development,  to  the  extent  that  these  can  be  differentiated  (Gaver  & 
Srinivasan,  1970).   Underlying  the  entire  area  of  problem  selection  is  the 
question  of  how  to  do  it  rigorously. 
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Returning  to  the  testimony  before  the  Subcommittee,  the  most  critical 
questions  in  the  effective  utilization  of  Federal  laobratories ,  according  to 
Hornig,  were: 

(1)  The  choice  of  problems,  their  significance,  and  the  feasi- 
bility of  finding  solutions  through  research  and  development; 

(2)  the  creation  of  capabilities  in  the  laboratories  which  can, 
in  fact,  solve  the  most  difficult  problems;  and   (3)   the  trans- 
lation of  the  results  of  the  laboratories'  work  into  action  in 
either  the  public  or  private  sectors.   (Subcommittee,  1968;  p.  9.) 

Mission  orientation  in  government  laboratories  usually  means  that 

there  is  a  parent  agency  whose  mission  is  being  supported  by  the  laboratory. 

Hornig  (Subcommittee,  1968;  p.  13),  for  example, has  said: 

An  effective  R&D  program  involves  a  dynamic  give  and  take  be- 
tween the  laboratory  and  its  parent  agency.   It  must  not  only 
carry  out  assigned  tasks,  but  spell  out  the  tasks  which  need 
to  be  performed;  it  must  be  a  source  of  ideas  for  its  parent 
agency  and  help  the  agency  to  put  the  laboratory's  output  into 
practice.   All  of  this  requires  a  very  close  identification 
between  a  laboratory  and  its  sponsoring  agency. 

Mission  orientation  in  R&D,  on  the  other  hand,  may  have  both  positive 
and  negative  aspects  (Nichols,  1971).   The  positive  aspects  of  this  rela- 
tionship were  explained  in  the  Bell  report  (Bureau  of  the  Budget,  1962)  as 
follows: 

Federal  operations,  such  as  the  governmental  laboratory,  enjoy 
a  close  and  continuing  relationship  to  the  agency  they  serve 
which  permits  maximum  responsiveness  to  the  needs  of  that  agency. 
Such  operations  accordingly  have  a  natural  advantage  in  con- 
ducting research,  feasibility  studies,  developmental  and  ana- 
lytical work,  user  tests  and  evaluations  which  directly  support 
the  management  functions  of  the  agency.   Furthermore,  an  agency- 
operated  research  and  development  installation  may  provide  a 
useful  source  of  technical  management  personnel  for  its  sponsor. 

However,  Hornig,  speaking  before  the  subcommittee,  also  admitted  that 

almost  all  reports  were  in  agreement  that  there  is  generally  excessive 

administrative  control  and  not  enough  freedom  given  to  the  directors  of 

laboratories.   He  attributed  this  not  only  to  civil  service  or  other 
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rules  but  to  past  practices  and  procedures.   He  mentioned  specifically 

as  a  difficult  problem  the  relationship  between  military  and  civilian 

personnel  in  the  direction  of  DOD  laboratories,  a  problem  which  was 

accentuated  by  the  policy  of  military  rotation  applied  indiscriminately 

to  technically  qualified  officers. 

The  difficulty  in  handling  parent  agency-laboratory  relationships 

in  a  beneficial  way  and  the  persistence  of  problems  in  this  area  were 

commented  upon  by  Glass  (1970,  p.  25)  in  reference  to  the  findings  and 

recommendations  of  the  Blue  Ribbon  Defense  Panel  issued  on  1  July  1970. 

After  almost  nine  years  of  a  concerted  effort  to  improve 
the  climate  and  performance  of  in-house  laboratories,  the 
findings  of  the  Blue  Ribbon  Defense  Panel  are  somewhat 
depressing.   Some  people  believe  that,  as  long  as  in-house 
laboratories  are  organizationally  imbedded  deep  within 
the  Military  Departments,  efforts  to  achieve  long-term 
improvements  in  laboratory  operations  are  sure  to  fail. 
They  feel  that  there  is  no  way  to  protect  the  laboratories 
from  the  staggering  overload  of  bureaucratic  red  tape 
and  from  diffuse,  fragmented  middle  management  levels 
that  are  apparently  unable  to  delegate  needed  authority. 
On  the  other  hand,  many  people  believe  that  the  close 
interaction  of  laboratories  with  their  sponsors  and 
their  Department's  needs  would  be  unnecessarily  perturbed 
if  new  organizational  barriers  were  created. 

The  preceding  discussion  on  the  mission  orientation  of  Federal 
laboratories  and  their  parent  agency  relationships  have  involved  the  area 
of  laboratory  controls.   That  is,  the  mission  orientation  necessitates 
selectivity  in  problems  to  be  investigated,  while  negative  aspects  of 
the  parent  agency-laboratory  relationships  are  reminiscent  of  the  dys- 
functional aspects  of  pyramidal  organizations  described  by  Argyris.   Are 
there  characteristics  of  Federal  laboratories  that  are  also  similar  to 
the  financial  responsibility  pressure  mentioned  by  Hall  and  Lawler? 
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One  possible  means  of  exerting  pressure  of  this  nature  was  vol- 
unteered by  Dr.  William  B.  McLean,  Technical  Director,  Navy  Undersea 
Warfare  Center,  to  the  Subcommittee  (1968).   He  felt  that  competition 
among  laboratories  in  given  areas  was  necessary  so  that  the  effectiveness 
of  the  laboratories  could  be  demonstrated  on  a  comparative  basis.   The 
process,  he  thought,  would  provide  high  incentives,  high  motivation,  and 
the  "only  nonsubjective  measure  of  effectiveness  in  R&D  .  .  .  ."  (Sub- 
committee, 1968;  p.  80).   Daddario,  however,  suggested  that  it  would  be 
difficult  to  implement  such  a  plan  of  duplicate  or  overlapping  laboratory 
facilities  so  that  Congress  could  effectively  deal  with  it  in  the  final 
determination  and  allocation  of  funds.   McLean  suggested  that  the  situa- 
tion actually  existed  in  some  areas — for  example,  the  Air  Force  probably 
provided  much  competition  to  NASA. 

McArthur,  commenting  on  McLean's  suggestion,  took  a  different 
approach.   He  remarked  that  competition  for  funds  exists  within  the  DOD 
laboratory  structure  and  that  this  runs  the  gamut  from  no  competition 
for  funds  to  laboratories  that  are  almost  totally  competitive  for  funds. 
Noting  that  the  latter  tends  towards  laboratories  becomming  "job  shops" 
with  few  or  no  longer  range  programs,  he  added  that  the  optimum  is  some- 
where between  the  two. 

Mr.  E.  M.  Glass,  DoD  Assistant  Director  for  Laboratories,  who 
was  accompanying  McArthur,  commented  that  a  laboratory  should  have  to 
compete  for  at  least  25  percent  of  its  funds  to  stay  healthy.   The  80  to 
90  percent  of  their  funding  that  some  Navy  laboratories  were  competing 
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for  from  various  customers  (at  that  time)  was  thought  to  be  excessive. 
He  observed  also  that  the  situation  with  the  Air  Force  laboratories, 
which  received  "block  funding"  and  were  not  dependent  at  all  on  customers, 
could  be  made  healthier  if  they  had  to  compete  for  some  of  their  funds. 
Competing  for  funds  from  a  customer,  he  explained,  resulted  in  a  healthier 
situation  by  coupling  laboratories  more  closely  with  their  customers. 

Thus,  it  is  obvious  that  pressures  not  unlike  "contract  pressures" 
experienced  in  the  non-Government  laboratories  also  exist  in  the  Federal 
laboratory  structure.   Accordingly,  the  organizational  committment  of 
scientists  should  be  a  characteristic  of  Federal  as  well  as  civilian 
laboratories.   That  this  is  a  characteristic  of  effective  Federal  labora- 
tories has  already  been  mentioned  in  some  of  the  comments  reported  above. 

The  general  environment  of  Federal  laboratories,  when  it  has  been 
bad,  has  been  due  to  weakness  in  the  areas  of  pay  and  professional  benefits, 
lack  of  challenging  assignments  and  personnel  recognition,  and  the  arbi- 
trary use  of  bureaucratic  regulations  with  resulting  frustrations  and 
inhibiting  effects  (Glass,  1970).   To  this  list  might  be  added  inadequate 
on-the-job  support,  as  reported  by  the  standing  committee  to  the  Federal 
Council  on  Laboratories.   The  standing  committee  also  stated  that  such 
things  as  job  titles,  the  retention  of  rights  to  patents,  consulting  fees 
and  honoraria,  security  matters,  and  freedom  to  consult,  lecture,  and 
teach  were  considered  relatively  unimportant  by  the  vast  majority  of 
1,025  Federal  scientists  who  were  respondents  to  a  questionnaire.   (Sub- 
committee, 1968). 
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Characteristics  of  a  desirable  organizational  environment  that 

would  maximize  effectiveness  were  prepared  by  a  Bureau  of  the  Budget 

committee  in  conjunction  with  the  Bell  report  (Bureau  of  the  Budget ,1962) . 

The  basic  requirements  for  such  an  organizational  climate  were  (Glass, 

1970,  p.  33  and  Appendix  IV): 

Reputation  and  atmosphere 

Excellence  of  staff  and  its  direction 

Importance  of  facilities  and  support 

Clarity,  challenge  and  urgency  of  objectives 

Maintenance  of  professional  caliber  of  operations 

Professional  leadership  by  professionals 

Professional  recognition  and  encouragement 

Encouragement  and  support  of  freedom  of  inquiry  and  method 

Advancement  on  merit 

Absence  of  unproductive  regulations  and  reports 

C.   DIMENSIONS  OF  THE  SCIENTIST-LABORATORY  INTERFACE 

The  preceding  analysis  dealt  with  laboratories  in  general  and 
more  specifically  with  Federal  laboratories.   Some  factors  tending  to 
enhance  or  inhibit  the  effectiveness  of  these  laboratories  were  identified 
and  discussed.   When  R&D  laboratories  are  considered  in  the  aggregate,  it 
may  be  possible  and  even  advantageous  to  think  of  individual  laboratories 
as  basic  entitites  with  characteristics  and  "personalities"  of  their  own. 
From  the  standpoint  of  assessing  the  relative  effectiveness  of  laboratories, 
this  method  may  also  be  appropriate.   In  fact,  a  DoD  program  is  in  progress 
that  takes  this  approach  (Glass,  1969).   Individual  laboratories  are  being 
ranked  on  the  basis  of  their  effectiveness  or  excellence  and  the  relation- 
ships of  various  laboratory  characteristics  to  this  measure  of  relative 
excellence  are  being  examined.   On  the  other  hand,  when  one's  concern  is 
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in  assessing  the  absolute  effectiveness  of  a  specific  laboratory  and  also 
involves  administrative  and  managerial  approaches  for  upgrading  that 
laboratory's  capabilities,  he  must  look  for  different  items  of  information. 
He  must  look  below  the  overall  laboratory  level  to  that  area  of  dynamic 
interaction  between  the  characteristics  of  the  laboratory  and  the  char- 
acteristics of  its  individual  professionals  who  man  the  laboratory.   This 
level  might  be  called  the  level  of  the  scientist-laboratory  interface. 

The  characteristics  of  a  laboratory  might  include  such  factors  as 
its  geographic  location,  whether  it  is  located  separately  or  within  a 
larger  military  complex,  overall  size,  budget  and  source  of  funds,  sub- 
ject area  of  research,  location  in  the  organizational  chain  of  command, 
position  on  the  research  or  developmental  end  of  the  R&D  continuum,  bal- 
ance between  in-house  and  contracted-out  research,  type  and  degree  of 
external  control  exerted  on  the  laboratory,  and  so  on.   These  variables 
may  be  important  in  the  comparative  evaluation  of  laboratories.   At  least, 
they  provide  convenient  and  reasonably  reliable  categories  for  describing 
individual  laboratories.   But  when  one  is  dealing  with  one  laboratory, 
many  of  these  variables  become  constants  which  are  of  interest  only  to 
the  extent  that  they  define  the  boundary  conditions  within  which  the 
laboratory  must  operate. 

On  the  other  side  of  the  interface,  a  great  deal  of  research  has 
been  conducted  regarding  the  characteristics,  attitudes,  and  motivations 
of  individual  professionals  in  R&D.   In  the  Hall  and  Lawler  (1971)  studies, 
for  example,  satisfaction  of  professionals  surveyed  was  measured  on  a 
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hierarchy  of  human  needs.   Their  questionnaire  yielded  scores  for  security, 
social  esteem,  autonomy,  self-fulfillment,  and  a  composite  score  which 
was  a  single  average  of  the  part  scores.   Job  involvement  was  measured  by 
an  attitude  scale.   As  in  the  Hall  and  Lawler  studies,  many  other  attempts 
have  been  made  to  find  behavioral  or  output  correlates  of  such  personal 
attributes.   These  output  criteria  might  be  mobility,  promotion,  or  some 
measure  of  volume,  quality,  and  influence  of  an  individual's  work.   But 
here  again,  in  the  case  of  a  specific  laboratory,  one  is  not  sure  whether 
the  previously  found  relationships  will  hold  between  the  personal  variables 
and  the  criterion  variables.   Why?   First,  effect  of  personal  variables,  such 
as  satisfaction  and  job  involvement,  on  output  are  dependent  on  other 
characteristics  of  the  individual.   For  example,  the  relationship  of  satis- 
faction to  some  measure  of  output  in  laboratories  may  depend  on  the  age 
distribution  of  the  laboratory's  scientists.   Second,  the  relationship  of 
the  personal  variables  to  the  outcome  variables  is  mediated  by  variables 
that  are  dimensions  of  the  work  or  laboratory  environment.   And  third, 
since  much  of  this  research  is  correlational,  it  is  often  difficult  or 
impossible  to  infer  the  causal  direction  of  the  relationships  established. 
For  example,  how  much  of  job  involvement  is  something  intrinsic  to  the 
individual  that  he  brings  to  the  job  and  how  much  the  result — the  output — 
of  his  experience  with  the  laboratory  work  environment? 

Accordingly,  when  interest  is  centered  on  a  specific  laboratory, 
the  crucial  relationships  are  at  the  level  of  the  scientist-laboratory 
interface.   Detailed  information  is  necessary  which  states  how  manipulation 


-19- 


of  the  alternatives  available  to  laboratory  management  affect  different 
scientists  to  bring  about  specific  outcomes  or  results.   It  is  especially 
important  to  discriminate  between  inputs  and  outputs.   The  laboratory's 
mission  and  tasks,  externally  imposed,  are  inputs  which  can  be  considered 
as  constants,  as  previously  explained.   The  scientist  and  what  he  brings 
to  the  laboratory  in  his  person  is  the  other  input  category.   The  labora- 
tory environment  or  climate  in  which  the  scientist  and  the  laboratory's 
organizational  structures,  procedures,  and  facilities  interact  might  be 
called  process.   The  result  of  the  interaction  is  output,  which  includes 
the  tangible  products  and  services  of  the  laboratory  and  also  such  secon- 
dary outcomes  as  experienced  manpower  and  changes  in  the  attitudes  and 
outlook  of  the  scientists.   Thus,  there  is  a  mechanism  for  change,  since 
the  input  (the  scientist)  will  have  been  changed  in  the  process  for  the 
next  iteration.    Similarly,  the  outcomes  will  provide  feedback  to  the 
laboratory  management  for  suggested  changes  in  the  operational  environment 
These  categories — input,  process,  output,  and  feedback — might  be  called 
stages.   They  are  schematically  diagrammed  in  figure  1. 


INPUT 


Missions 

Tasks 

Scientists 


A 


-^ 


PROCESS 


Laboratory  environment , 
"climate"  for  scientist 
x  laboratory  interaction 


OUTPUT 


h 


Products 
Services 

Experienced  manpower 
Attitude  change 


Feedback 


Figure  1.   Simplified  structure  and  dynamics  of  laboratory  operations, 
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The  stages  can  be  further  broken  down  into  dimensions  and  variables 
as  shown  in  figure  2.   Of  the  input  variables,  education  can  be  broken 
down  into  PhD,  non-PhD,  and  Engineer  categories.   Pelz  and  Andrews  (1966) 
used  these  categories  and  further  classified  their  subjects  into  (1)  PhD's 
in  development-oriented  laboratories,  (2)  PhD's  in  research-oriented 
laboratories,   (3)  non-PhD' s  in  development-oriented  laboratories  not 
dominated  by  PhD's  (called  "engineers")?   (4)  non-PhD's  in  PhD-dominated 
laboratories  (called  "assistant  scientists") ,  and   (5)  non-PhD  scientists 
in  research-oriented  laboratories  not  dominated  by  PhD's  (all  in  Govern- 
ment settings).   Age  is  a  continuous,  quantitative  variable  which  can  be 
categorized,  if  desired.   Orientation  is  an  individual's  preference  for 
research  or  development  activities.   Specialization  can  be  identified  by 
an  individual's  area  of  specializationand  further  subdivided  into  the 
number  of  subareas  in  which  he  feels  competent  to  do  work.   Professional 
style  refers  to  the  type  of  problems  and  the  approach  one  prefers.   The 
other  dimensions — attitude,  ability,  personality,  physical  makeup — are 
named  to  fill  out  the  list.   Numerous  other  variables  might  be  listed.   Of 
the  listed  dimensions,  education  and  age  are  parameters  that  can  be  most 
reliably  obtained  for  personnel  in  a  laboratory.   Orientation,  speciali- 
zation, and  professional  style  could  be  readily  obtained  by  some  form  of 
questioning  the  technical  staff.   The  others  are  more  complex,  more  diffi- 
cult to  assess,  and  more  unlikely  to  be  related  to  the  process  and  output 
variables  in  any  simple,  predictable  manner. 

The  process  variables  are  those  that  are  characteristic  of  the 
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procedures  and  working  environment  of  the  laboratory.   These  are  the 
elements  that  can  be  manipulated  to  change  the  output  variables.   Orien- 
tation refers  to  the  research  vs.  development  bent  of  the  laboratory. 
The  proportion  of  short  vs.  long-range  programs  could  be  another  variable 
in  this  dimension.   The  important  dimension  of  control  can  be  identified 
by  a  number  of  variables  such  as  loose  vs.  tight,  number  of  checkoffs 
required  for  decision  making,  degree  of  influence  assertable  by  the  indi- 
vidual scientist  in  decision  making,  and  sources  of  ideas  and  persons 
involved  in  -goal  setting.   The  time  span  of  discretion  might  be  another 
variable  in  the  control  area.   Work  unit  characteristics  can  be  defined 
by  the  degree  of  homogeneity  or  similarity  of  the  individuals  within  the 
unit  and  the  size  and  age  of  the  unit.   Individual  assignments  can  be 
classified  into  the  proportion  of  time  spent  in  R&D  work  vs.  adminis- 
trative tasks,  the  proportion  of  technical  time  spent  in  research  vs. 
development  activities,  and  the  number  of  R&D  functions  to  which  one  is 
assigned.   The  communication  structure  and  dynamics  of  the  laboratory 
can  be  defined  in  terms  of  the  number  of  contacts  an  individual  has  within 
his  own  work  unit,  the  number  of  contacts  he  has  outside  the  work  unit, 
and  the  frequency  of  these  contacts.   Rewards  can  be  looked  at  from  the 
traditional  aspects  of  promotion,  pay,  and  fringe  benefits;  they  might 
be  identified  by  some  less  tangible  forms  of  reinforcement,  such  as  re- 
cognition. 

The  problem  in  using  the  process  variables  in  any  analysis  is 
their  method  of  measurement.   On  one  hand,  many  values  could  be  determined 
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by  asking  the  management  and  by  going  through  organization  charts  and 
other  laboratory  documentation.   But  research,  and  the  practical  ex- 
perience of  many  persons,  has  shown  that  official  statements  and  the 
opinions  of  the  individual  workers  in  the  organization  may  vary  con- 
siderably.  Since  the  framework  of  analysis  described  above  involves  the 
interaction  between  the  individual  scientist  and  the  laboratory  operating 
environment,  it  would  probably  be  necessary  to  ask  the  scientist  to  scale 
most  of  these  variables  from  his  own  point  of  view. 

Turning  to  the  output  variables,  those  listed  under  product  or 
service  require  further  operational  definition  before  they  can  be  mean- 
ingfully evaluated.   This  problem  will  be  taken  up  in  the  next  section 
of  this  paper.   The  attitude  change  variables,  again,  would  have  to  be 
assessed  on  the  basis  of  solicited  individual  opinions. 

The  foregoing  list  of  variables  borrows  heavily  from  the  monu- 
mental Pelz  and  Andrews  (1966)  work  on  scientists  in  organizations;  their 
categorization  by  dimensions  and  stages  and  the  inferred  direction  of 
causation  do  not.   The  description  and  discussion  of  general  trends  in 
the  R&D  field,  and  especially  in  laboratories,  plus  the  detailed  break- 
down of  the  structure  and  dynamics  within  a  laboratory,  should  provide 
the  necessary  background  and  framework  for  analyzing  the  specific  prob- 
lems posed  in  this  study  and  making  appropriate  recommendations  for  their 
solution. 
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III.   EVALUATION  OF  LABORATORIES 

A.   PURPOSE,  NEED,  AND  APPROACHES 

Evaluation  of  the  effectiveness  of  laboratory  performance  is  a 

responsibility  of  those  who  manage  the  laboratories  and  the  parent  agencies 

that  support  them.   While  the  need — or  requirement — and  even  the  desire 

to  do  so  may  be  strong,  the  question  of  how  to  conduct  the  evaluation  is 

one  that  has  not  found  a  universally  acceptable  answer.   Even  the  question 

of  "why"  may  not  find  a  convergence  of  answers.   Glass  (1969,  p.  1)  writes: 

R&D  managers  have  been  trying  for  many  years  to  devise 
better  yardsticks  to  measure  the  effectiveness  and  utility 
of  laboratories.   Most  managers  agree  that  such  techniques 
are  needed,  but  few  can  agree  on  how  such  appraisals  can  or 
should  be  made. 

The  evaluation  of  laboratories  was  a  question  of  prime  importance 
to  the  Subcommittee  on  Science,  Research,  and  Development  as  an  aspect  of 
fostering  the  best  utilization  of  Federal  laboratories.   Most  of  the  wit- 
nesses before  the  Subcommittee  were  asked  to  comment  upon  their  philosophy 
and  approaches  to  laboratory  evaluation.   In  addition,  the  witnesses  were 
asked  to  submit  answers  to  written  questions  provided  them  following  their 
appearance  before  the  Subcommittee.   One  of  the  questions  dealt  with  the 
problem  of  evaluation.   The  views  expressed  in  their  statements  provide 
excellent  insight  into  the  mechanisms  of  evaluation  and  the  philosophy  be- 
hind the  approaches  taken. 

Hornig  (Subcommittee,  1968;  p.  22),  provided  a  broad  overview  of 

the  appraisal  of  laboratories: 

Appraisal  of  in-house  laboratory  performance  is 
a  normal  responsibility  of  agency  management.  .  .  . 
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Procedures  in  use  in  the  Government  .  .  .  involve  such 
techniques  as  visits  to  the  laboratory  by  teams  of  agency 
management  representatives;  evaluation  of  results  by 
agency  management  and — especially  where  more  basic  re- 
search is  involved — by  outside  advisory  groups;   and 
continuing  reviews  of  laboratory  operations  through  re- 
ports, audits,  conferences,  day-to-day  contacts,  and 
so  on.   In  some  cases,  development  activities  lend  them- 
selves to  controlled  scheduling  procedures  such  as  PERT, 
but  such  control  methods  are  generally  not  applicable  to 
research  near  the  basic  end  of  the  spectrum.   Evaluations 
are  necessarily  qualitative  rather  than  quantitative  to 
a  considerable  degree,  and  involve  judgments  based  on 
such  factors  as  experience  and  comparison  with  good 
practice  elsewhere. 

The  prime  objective  is  not  the  application  of  any 
specific  set  of  administrative  techniques,  but  the  eleva- 
tion of  the  quality  and  efficiency  of  administration  of 
Federal  laboratories  totally,  including  such  matters  as 
maintenance  of  challenging   and  relevant  laboratory  missions, 
elevation  of  salary  scales  to  attract  first  class  managers, 
and  securing  sufficient  freedom  for  laboratory  managers. 

McLean  (Subcommittee,  1968;  p.  84)  brought  out  the  difficulty  of 

finding  a  proper  criterion  to  measure  in-house  laboratory  effectiveness 

that  could  be  compared  with  the  profit-loss  ledger  of  the  profit-oriented 

civilian  enterprises.   He  also  mentioned  the  need  for  subjective  appraisals 

and  review  committees,  as  follows: 

For  any  organization  or  individual  to  feel  success- 
ful there  must  be  some  mechanism  for  measuring  the  degree 
in  which  they  have  fulfilled  their  goals.   In  an  organiza- 
tion which  is  profit-oriented,  such  an  evaluation  is  straight- 
forward, rigorous,  and  simple.   If  the  figures  are  in  the 
black,  all  associated  with  the  organization  are  happy.   If 
they  are  in  the  red,  or  tending  toward  the  red,  then  some- 
thing must  be  done  to  rectify  the  situation.   Government 
organizations,  military  organizations,  educational  institu- 
tions, and  research  and  development  activities,  whenever 
they  are  adequately  removed  from  the  profit-making  pressures, 
have  a  more  difficult  time  in  establishing  a  proper  evaluation 
of  the  effectiveness  of  their  processes  and  results.   For 
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all  such  organizations  I  believe  the  evaluation  must 
be  on  the  basis  of  competition  similar  to  that  in- 
volved in  making  a  profit.   The  fact,  however,  that 
results  cannot  easily  be  expressed  in  terms  of  a 
single  variable,  such  as  money,  tends  to  make  the 
evaluation  process  much  more  difficult.   Governments 
are  judged  by  history,  and  military  organizations  by 
wars.   These  are  very  harsh  and  final  judgments  and 
do  not  provide  a  very  adequate,  self-rectifying  mech- 
anism. 

In  essence,  the  appraisal  of  contractor  and 
laboratory  performance  is  limited  by  the  capabilities 
of  the  individuals  available  to  perform  the  appraisal. 
Of  necessity,  an  appraiser  must  be  a  person  who  has 
been  very  successful  in  the  field  being  evaluated. 
Yet,  every  appraiser  has  his  own  set  of  biases  and 
believes  that  his  own  approach  is  the  only  correct 
approach.   The  competitive  system  is  the  only  appraisal 
system  that  leaves  open  the  possibility  of  innovation. 

In  spite  of  these  difficulties  in  evaluation,  the 
Navy  is  setting  up  technically  competent  review  commit- 
tees to  review  laboratory  performance.   The  effect  of 
these  committees  on  laboratory  performance  has  yet  to  be 
evaluated. 

In  his  testimony  before  the  Subcommittee,  Astin  acknowledged  the 
need  to  have  some  mechanism  for  "rating"  the  many  laboratories  within  his 
organization,  the  National  Bureau  of  Standards,  and  implied  that  the  pur- 
pose of  such  ratings  was  to  do  something  about  strengthening  ineffective 
management  and  leadership  and  to  encourage  those  that  were  strong.   In 
his  written  comments,  however,  he  presented  (Subcommittee,  1968;  p  69)  an 
excellent  discussion  of  the  mechanism  of  program  review  as  an  evaluative 
process : 

.  .  .   An  annual  or  sometimes  more  frequent  series  of 
reviews  of  all  of  the  significant  programs  of  the  Bureau 
are  made  by  operating  personnel  to  top  NBS  officials. 
Generally  programs  are  closely  related  to  one  or  several 
organizational  units.   The  program  of  each  division  is 
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also  subject  to  review  at  least  annually  by  Advisory  Com- 
mittees of  the  National  Academy  of  Sciences.   Finally,  an 
outside  evaluation  of  the  Bureau  as  a  whole  is  made  by  a 
Statutory  Visiting  Committee  which  reports  annually  to  the 
Secretary  of  Commerce  on  the  efficiency  of  NBS  operations. 

At  the  program  reviews  those  responsible  for  the 
program  discuss  their  past  accomplishments,  present  work 
and  future  program  plans.   Information  is  given  and  ques- 
tions raised  concerning  the  adequacy  and  competence  of  the 
staff  and  future  requirements;  similar  matters  are  covered 
in  respect  to  facilities,  equipment,  funds  and  other  re- 
sources.  Program  requirements  and  priorities  are  discussed 
in  respect  to  their  relationship  to  national  needs.   On  the 
basis  of  all  of  the  above,  the  present  health  and  future 
outlook  of  Bureau  programs  are  assessed  by  NBS  officials, 
priorities  are  set,  and  appropriate  allocations  and  commit- 
ments are  made.   Occasionally  decisions  are  made  that  some 
programs  of  substantially  diminished  importance  have  out- 
lived their  usefulness.   When  such  decisions  are  made,  the 
staff  members  involved  are  reassigned  when  possible  or  re- 
duced in  force  when  reassignment  is  impractical. 

Other  speakers  before  the  Subcommittee  also  referred  to  project 
and  program  reviews  and  evaluation  panels  and  advisory  groups  as  mechanisms 
for  conducting  the  assessment  of  laboratory  effectiveness.   Some,  particu- 
larly the  NASA  representative,  emphasized  continuous  and  close  technical/ 
professional  communications  between  the  headquarters  and  the  laboratory  as 
a  means  of  appraising  laboratory  performance.   The  comments  of  MacArthur 
(Subcommittee,  1968;  p.  151)  were  of  particular  relevance  to  this  study: 

When  we  evaluate  the  quality  of  a  laboratory  there 
are  many  criteria  we  use.  The  three  services,  the  Air 
Force,  the  Army,  and  the  Navy,  have  advisory  groups  which 
periodically  review  the  programs  of  the  laboratories  and 
come  up  with  recommendations  in  terms  of  whether  they  are 
above  standards,  below  standard,  fine,  mediocre,  whatever 
they  might  be. 

We  also  look  at  whether  they  have  meaningful  missions. 

Secondly,  within  the  Department  of  Defense  at  the 
D.D.R.  &  E.  level  where  I  come  from,  we  look  at  programs 
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from  a  programmatic  standpoint,  from  a  technical  stand- 
point, and  at  that  point  we  look  at  the  contributions 
the  laboratories  are  making  to  that  program. 

Thirdly,  as  you  no  doubt  know,  some  of  our  labor- 
atories are  involved  at  only  one  end  of  the  R.  &  D . 
spectrum,  research  and  technology.   Other  laboratories 
are  involved  throughout  the  whole  R.&D.  spectrum  through 
engineering  development  and  test  and  evaluation. 

Now,  one  of  the  criteria  we  use  when  a  laboratory 
is  involved  at  the  research  and  technology  end  of  the 
spectrum  is  how  much  of  their  output  over  the  last  few 
years  has  been  incorporated  in  some  of  our  systems  devel- 
opment programs . 

In  a  laboratory  that  is  involved  in  engineering 
development,  we  look  at  the  effectiveness  of  the  systems 
or  hardware  they  have  developed  or  managed,  and,  lastly, 
we  look  at  the  individual  laboratory  director's  inde- 
pendent research  program  and  look  at  how  he  has  managed 
his  funds,  what  he  has  done,  where  he  has  invested  them, 
but  the  real  test  is  how  much  business  he  got  based  on 
those  investments  he  has  made. 

Among  the  supplemental  questions  addressed  to  MacArthur,  was  one 
that  asked  him  to  explain  the  DoD  system  for  evaluating  contractors  that 
provide  R&D  and  to  state  to  what  extent  the  system  should  or  could  be  used 
in  appraising  the  work  of  the  in-house  laboratories.   MacArthur,  in  his 
reply,  described  the  DoD  Contractor  Performance  Evaluation  (CPE)  Program, 
a  system  designed  to  provide  an  orderly  and  uniform  technique  of  determin- 
ing and  recording  the  effectiveness  of  contractors  primarily  for  hardware 
development  and  production.   In  evaluating  the  appropriateness  of  the  CPE 
procedure  for  in-house  laboratory  appraisals,  MacArthur 's  comments  (Sub- 
committee, 1968;  p.  180)  were  particularly  illuminating: 

For  those  contracts  where  the  end  product  is  new 
technology  or  new  scientific  findings,  CPE  can  be  utilized, 
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if  the  Military  Departments  consider  it  desirable.   How- 
ever, a  less  formal  evaluation  of  such  contracts  is  gen- 
erally utilized.   This  is  usually  in  the  form  of  a  sub- 
jective appraisal  by  the  project  monitor. 

The  CPE  System  appears  to  be  most  suitable  for 
evaluating  specific  programs  for  which  there  are  meaning- 
ful performance  standards  and  mileposts.   I  might  add  that 
similar  criteria  are  utilized  to  evaluate  the  hardware 
development  laboratories,  although  in  a  different  form. 
In  other  words,  the  CPE  System  is  designed  to  evaluate 
performance  on  a  specific  well-defined  project  or  program. 
Its  purpose  is  not  to  evaluate  over-all  organizational 
effectiveness . 

Much  of  the  work  of  Defense  laboratories  is  in  the 
areas  of  long-range  research  and  technology.   Thus,  a 
system  such  as  CPE  would  not  be  generally  applicable. 
One  advantage  we  see  in  the  establishment  of  military  prob- 
lem oriented  weapon  centers  is  that  the  utility  of  their 
output  can  be  measured  fairly  directly  on  a  real-time 
basis.   Practically  everyone  knows  and  can  measure  the  tre- 
mendous productivity  of  a  NOTS ,  China  Lake.   Its  output 
goes  directly  into  the  military  inventory.   On  the  other 
hand,  the  output  or  product  of  a  research  laboratory  is 
much  more  difficult  to  assess.   Many  years  may  pass  before 
the  utilization  of  new  science  or  technology  can  be  meas- 
ured meaningfully . 

The  principal  method  used  for  Defense  laboratories 
is  peer  rating  or  evaluation,  either  by  in-house  people, 
those  on  the  outside,  or  combinations  of  both.   This  is 
only  part  of  the  story,  however.   Program  evaluation  in 
terms  of  need,  priority,  technical  content,  and  similar 
factors  probably  have  a  greater  bearing  upon  the  appraisal 
of  laboratories  than  direct  institutional  evaluation. 
Through  program  evaluation,  one  usually  makes  decisions 
on  resource  allocation  which  ultimately  determine  the  future 
of  that  laboratory  responsible  for  the  program  execution. 

The  comments  to  the  Subcommittee  extracted  above  were  representative 

of  the  others  who  testified.   The  needs  and  purposes  of  the  evaluations 

seemed  to  fall  under  the  following  categories: 

Determine  the  quality  and  efficiency  of  the  administration 
Evaluate  the  meaningf ulness  of  programs 
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Determine  the  health  and  future  outlook  of  programs 
Look  for  challenging,  relevant  missions;  adequate 

pay  of  professional  staff;  and  freedom  for  the 

laboratory  manager 
Set  priorities  and  determine  allocation  and  commit- 
ment of  funds 
Look  for  laboratories  and/or  programs  that  have 

served  their  usefulness  to  shut  them  down 
Shore  up  the  management  and  leadership  where  necessary; 

give  encouragement  to  good  leadership 
Provide  guidance  to  the  laboratories 
Determine  adequacy  and  competence  of  staff  and  future 

requirements 
Determine  adequacy  of  facilities,  equipment,  funds, 

and  other  resources. 

They  all  tended  to  agree  that  evaluation  of  the  research  and  tech- 
nology end  of  the  R&D  spectrum  is  the  difficult  problem,  that  this  has  to 
be  done  subjectively/qualitatively,  and  that  some  sort  of  panel  of  experts 
is  an  appropriate  mechanism.   There  was  an  underlying  current  that  program 
evaluation  and  institutional  evaluation  are  different  but  that  program  re- 
views over  a  period  of  time  can  serve  the  function  of  institutional  reviews. 
Perhaps  the  distinction  between  program  reviews  and  institutional  reviews 
lies  in  the  criteria  available  for  evaluation.   Program  reviews  have  more 
specific  goals  against  which  progress  towards  those  goals  can  be  evaluated. 
Institutional  reviews,  however,  emphasize  such  intangibles  as  overall  quality, 
relevance  of  work  attempted,  and  technical  competence.   If  this  were  so, 
then  the  distinction  between  program  reviews  and  institutional  reviews  might 
be  one  of  performance  vs.  quality.   It  should  be  noted  that  good  performance 
on  the  part  of  a  laboratory  does  not  necessarily  ensure  effectiveness  or 
quality  in  the  product,  even  in  hardware  systems.   (This  matter  will  be 
further  elaborated  under  the  discussion  of  evaluation  criteria.) 
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B.   ADVISORY  PANELS 

Since  the  use  of  advisory  panels  seems  to  be  generally  accepted  as 
a  method  for  evaluating  laboratories,  a  more  detailed  examination  of  the 
process  would  seem  to  be  in  order.   While  the  panels  are  usually  termed 
advisory,  the  term  may  mean  anything  from  the  simple  lack  of  directive 
authority,  at  one  end,  to  advisory  in  the  literal  sense  of  the  word  at  the 
other.   Usually,  the  distinction  is  closely  associated  with  the  level  of 
the  convening  authority  of  the  panel,  tending  to  be  more  closely  allied  to 
direction  monitoring,  and  decision-making  processes  when  it  is  formed  by, 
and  reports  to,  a  headquarters  above  the  laboratory.   Another  dimension 
underlying  the  panel  system  is  the  constitution  of  the  panel — that  is,  the 
source  and  types  of  persons  making  up  the  panel.   And  finally,  closely  re- 
lated to  these  two  dimensions,  is  that  of  the  basic  purpose  of  a  panel. 

The  following  interchange  between  Daddario  and  MacArthur  at  the 
Subcommittee's  (1968,  p.  152)  hearings  highlights  the  use  of  advisory 
panels  at  the  highest  levels  in  the  services: 


Mr.  Daddario.   You  talk  about  judgment  by  peers.  Who 
are  they  and  how  do  you  get  them  together? 

Dr.  MacArthur.   There  are  three — let  me  address  each 
service.   In  the  Air  Force  we  have  a  board  of  advisers  which 
is  a  panel  of  the  scientific  advisory  board  and  they  are 
from  the  outside.   They  involve  individuals  from  the  indus- 
trial world,  individuals  from  the  universities,  and  from 
nonprofit  organizations. 

In  the  Navy  we  have  the  technical  evaluation  board 
which  is  a  part  of  the  Naval  Research  Advisory  Council. 

Again,  all  members  of  this  advisory  board  or  technical 
evaluation  group  are  from  outside  the  Department  of  Defense. 

The  Army  is  different  in  that  they  have  a  group  which 
is  composed  solely  of  all  in-house  individuals  and  they  per- 
form an  appraisal  every  3  years.   We  call  it  the  triannual 
survey  group. 
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These  are  the  advisory  boards  I  referred  to. 

Mr.  Daddario.   How  are  the  laboratory  directors  in- 
volved in  the  evaluation,  if  at  all? 

Dr.  MacArthur.   Well,  these  advisory  boards,  they  do 
more  than  read  papers.   They  actually  visit  the  labs.   They 
talk  to  the  laboratory  directors  and  the  key  people  at  the 
laboratories  to  see  what  they  are  involved  in,  what  they 
are  doing,  what  their  mission  is,  what  they  have  contributed 
in  the  last  year,  and  what  they  intend  to  work  on  in  the 
following  years,  and  why. 

One  of  the  biggest  problems  we  have  been  having  in  the 
past  years  was  that  some  of  these  laboratories  weren't  in- 
volved in  important  questions.   That  is  one  of  the  things 
we  have  been  stressing,  that  they  have  to  get  more  involved 
in  important  military  problems. 


Here,  while  there  is  obviously  interaction  between  the  panels  and 
the  laboratories,  the  purpose  of  the  panels  seems  generally  to  be  to  advise 
the  responsible  command  above  the  laboratory  level  and  to  encourage  the 
implementation  of  the  command's  policies — e.g.,  become  more  involved  in  rel- 
evant military  problems.   The  evaluative  nature  of  the  panel  is  especially 
evident  in  the  Army's  case  by  its  constitution.   On  the  other  hand,  the 
following  exchange  between  Daddario  and  Astin  (Subcommittee,  1968;  p.  50) 
very  definitely  implies  a  panel  that  is  responsive  at  the  laboratory  direc- 
tor's level  and  serves  at  his  discretion. 


Mr.  Daddario.   How  do  you  bring  the  individuals  to- 
gether and  how  do  you  allow  the  people  who  are  involved  in 
the  Federal  laboratories  to  participate?  Yesterday  we  had 
some  testimony  on  the  need  to  do  this  more  often  than  we 
do,  not  only  because  you  can  get  different  points  of  view, 
but  also  because  laboratory  directors  would  find  out  more 
what  was  going  on  and  they  could  improve  their  own  manage- 
ment activities. 

Dr.  Astin.   Well,  it  is  my  feeling  that  every  labora- 
tory manager  or  director  should  have  some  responsibility 
for  formulating  at  least  a  portion  of  his  program.   In  gen- 
eral the  laboratory  supports  the  mission  of  the  agency,  and 
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to  be  effective,  it  has  to  be  responsive.   But  if  the 
program  is  to  be  dynamic  and  effective,  then  formulation 
of  some  portion  of  this  program  has  to  be  under  the  con- 
trol of  the  laboratory  manager. 

At  the  same  time  it  is  desirable,  I  think,  for  him 
to  have  mechanisms  for  seeking  advice  from  experts  in  the 
technologies  involved.   In  general  advisory  committees  to 
laboratory  managers  have  proved  very  helpful. 

The  discussion  in  section  A,  above,  also  emphasized  the  role  of 
panels  in  program  formulation  and  evaluation.   One  of  the  "purest"  examples 
of  the  use  of  panels  for  such  purposes  is  the  practice  in  some  Federal 
agencies  to  assemble  panels  of  specialists  to  evaluate  proposals  for  Federal 
support.   The  problem  that  may  arise  when  scientists  themselves  are  used 
as  the  panel  members  is  explained  by  Nichols  (1971,  p.  33): 


Some  representatives  and  senators  have  been  asking 
searching  questions  about  what  they  see  as  an  overelaborate 
"buddy  system"  masquerading  as  a  scientific  "judicial  system". 
All  too  few  people  in  Congress  understand  the  enormous,  largely 
unpaid  efforts  invested  by  scientists  in  the  panel  review  sys- 
tem used  by  most  agencies  to  evaluate  proposals  for  federal 
support.   Worse  still,  the  dedication  of  most  of  these  review- 
ers to  high  quality  is  just  not  grasped  by  a  number  of  influ- 
ential members  and  staff  of  congressional  committees.   Log- 
rolling, it  seems  to  some  congressmen,  is  being  increasingly 
substituted  for  what  were  once  rigorous  standards  of  scientific 
quality  and  productivity. 

********* 

Despite  the  valiant  and  often  insightful  efforts  of  a 
few  scientist-analysts,  Congress  sees  little  progress  being 
made  in  solving  the  difficult  "apples  and  oranges"  problem  of 
allocating  resources  to  and  within  various  federal  R&D  areas. 


Nichols'  comments  suggest  that  advisory  panels  play  an  important 
decision-making  role  in  the  Federal  R&D  effort.   When  this  is  the  purpose 
of  a  panel,  Gideonse  (1970)  adds  another  dimension  to  the  use  of  panels 
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for  program  evaluation.   He  maintains  that  the  scientist-panel  approach 
to  decision  making  is  only  proper  for  nan-mission-oriented  agencies  and 
those  supporting  fundamental  research.   But  when  agencies  have  more  spe- 
cific missions,  persons  selected  to  serve  in  an  advisory  capacity  must 
have  the  expertise  required  to  make  the  decisions  involved,  especially  of 
a  policy  setting  nature. 

Thus,  the  use  of  advisory  panels  in  the  management  of  R&D  activities 
is  a  complex  procedure  involving  at  least  a  consideration  of  the  purpose 
of  the  panel,  the  agency  that  it  supports,  and  who  shall  constitute  the 
panel.   Finally,  as  mentioned  in  section  II  A,  top-level  committees  formed 
as  "overseers"  of  research  may  well  be  a  symptom  of  deterioration  in  the 
R&D  process  and  not  a  cure  (Argyris,  1968). 

C.   CRITERIA 

Criteria  to  be  used  for  evaluating  laboratories  are  closely  linked 
to  the  purpose  of  the  evaluation.   In  addition,  criteria  are  of  little 
worth  if  there  is  no  way  to  provide  measurements  which  indicate  a  position 
on  the  criterion  for  a  given  laboratory.   Thus,  such  terms  as  measures  of 
effectiveness,  figures  of  merit,  cost/benefit  ratios,  and  so  forth  arise 
to  fill  this  need.   Unfortunately,  the  possibility  of  finding  appropriate 
measurements — "yardsticks,"  according  to  Glass — become  more  remote  as  the 
purpose  of  the  evaluation  becomes  global  and  abstract.   In  fact,  it  becomes 
difficult  just  to  find  appropriate  criteria,  much  less  methods  for  measure- 
ment, when  such  is  the  case.   Take,  for  example,  the  desire  to  determine 
the  technical  excellence  of  a  laboratory.   What  criteria  does  one  use  to 
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to  do  this?   And  what  measurements  are  to  be  taken  to  denote  the  position 
of  a  laboratory  on  the  criterion? 

One  approach  is  to  look  for  intermediate,  more  readily  quantifiable 
criteria  as  estimates  of  the  more  general,  abstract  criteria.   This  approach 
assumes,  of  course,  that  the  relationships  of  the  intermediate  or  part 
criteria  to  each  other  and  to  the  higher  level  criteria  are  known.   Usually 
this  is  not  the  case.   This  approach,  when  carried  to  the  extreme,  results 
in  measurements  of  manner  of  performance — or  simply,  performance — rather 
than  measures  of  quality  (Hatry,  1970).   Typically,  such  measures  involve 
quantity  of  output,  rate  of  production/output,  costs,  manpower  utilization, 
adherence  to  schedules,  adherence  to  standards  and  specifications,  and  so 
forth.   As  indicators  of  workload  and  managerial  efficiency  the  measures 
may  be  appropriate.   A  dramatic  example  of  the  meaninglessness  of  such 
measures  involves  the  gross  national  product  (GNP) ,  presumably  an  indicator 
of  the  quality  of  life  in  the  nation.   A  family  makes  a  contribution  to  the 
GNP  when  its  mother  dies:   the  GNP  goes  up  because  more  expenditures  are 
necessary  to  procure  the  services  which  the  mother  formerly  provided  for 
nothing  (Clausen,  1971).   A  similar,  although  not  so  dramatic  an  example, 
can  arise  when  program  reviews  are  used  for  the  evaluation  of  a  laboratory 
and  progress  on  a  PERT  or  GANTT  chart  is  used  as  the  criterion  of  the  quality 
or  utility  of  the  work  being  performed.   In  fact,  the  opposite  is  more 
likely  to  be  true:   quality  is  sacrificed  to  meet  the  milestones. 

Another  problem  in  evaluating  R&D  work  in  personnel  research  is  the 
fact  that  it  deals  with  human  beings  and  their  value  systems  (Gideonse, 
1971).   Consequently,  the  findings,  results,  and  recommendations  of  research- 
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even  the  topics  chosen  to  be  researched—are  evaluated  against  social  and 
political  criteria  of  a  very  general  nature  over  and  above  the  scientific 
quality  of  the  work.   To  find,  as  the  Navy  did  some  years  ago,  that  vol- 
unteers from  certain  areas  of  the  United  States  who  were  very  young  and 
high-school  dropouts  created  so  much  trouble  that  the  Navy  would  be  better 
off  not  enlisting  them  has  the  seeds  of  a  very  touchy,  social-political 
problem.   On  the  other  hand,  if  personnel  and  behavioral  science  research 
does  not  come  to  grips  with  socially  relevant,  politically  loaded  questions, 
it  will  be  criticized  for  skirting  the  problems  that  "really  count."   It 
is  understood  that  the  Taskforce  on  Manpower  of  the  Defense  Science  Board 
is  taking  a  very  hard  look  at  the  adequacy  of  Defense  manpower  research  in 
respect  to  the  pressing,  social-political  problems  that  involve  the  Depart- 
ment of  Defense.   The  problems  involved  in  finding  adequate  criteria  for 
research  and  program  evaluation  dealing  with  the  "quality  of  life"  are 
thoroughly  explored  by  Clausen  (1971)  and  Hatry  (1970) . 

Another,  and  often  unrecognized,  area  creating  difficulty  in  assess- 
ing the  effectiveness  of  personnel  research  and  human  factors/man-machine 
R&D  requires  mention.   This  is  the  fact  that  the  absence  of  problems  in  a 
completed  system  is  a  sign  of  good  work  in  these  disciplines.   For  example, 
an  operator  in  a  complex  system  may  find  information  displayed  so  it  is 
readily  monitored;  controls  of  the  right  size,  shape,  and  location;  and 
work-space  and  environment  conducive  to  extended-duration  performance  in 
comfort.   His  training,  job  aids,  work  procedures,  and  workload  are  adequate 
He  has  ready  communication  channels  to  other  operators  or  superiors  when 
necessary.   Years  of  research  on  small,  apparently  insignificant  research 
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problems  have  all  contributed  to  the  knowledge  required  to  achieve  this 
situation.   An  experienced  and  talented  group,  thoroughly  conversant  with 
this  knowledge,  is  required  to  put  this  wealth  of  information  into  the 
system  design.   Yet  current  research  of  the  same  type,  looking  to  the  future, 
may  be  severely  criticized  by  the  unsophisticated  observer  as,  for  example, 
"analysis  of  variance"  research  on  insignificant  or  irrelevant  problems 
with  little  prospect  of  payoff.   It  is  indeed  a  difficult  problem  to  deter- 
mine the  payoff  or  effectiveness  of  long-term  research  at  or  near  the  basic 
science  end  of  the  R&D  spectrum. 

Returning,  again,  to  the  hearings  of  the  Subcommittee  on  Science, 
Research,  and  Development,  some  of  the  criteria  mentioned  by  those  testi- 
fying were  as  follows: 


1)  Comparison  of  a  laboratory  with  "experience" 

2)  Comparison  with  "good  practice" 

3)  How  much  of  the  output  of  research  and  technology  has 
been  incorporated  into  system  development  programs 

4)  Effectiveness  of  the  system  hardware  managed  by  the 
laboratory 

5)  Amount  of  subsequent  business  received  from  an  initial 
expenditure  of  independent  laboratory  research  funds 

6)  Peer  ratings 


The  majority  of  these  criteria  depend  on  judgment,  partly  or  completely,  in 
the  evaluative  process.   The  third  listed  has  often  been  called  "implementa- 
tion," and  is  largely  anecdotal.   A  direct,  simple  approach  to  implementing 
the  "implementation"  criterion  is  highly  likely  to  be  seriously  criticized 
as  in  the  case  of  Project  HINDSIGHT  (Radnor,  Rubenstein,  and  Tansik,  1970). 
Item  4  is  obviously  the  ultimate  criterion,  but  the  only  valid  measurement 
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is  performance  in  the  operational  environment.   Ignoring  the  difficulty 
in  doing  this,  there  is  an  added  problem  of  evaluating  partial  criteria 
for  laboratories  that  support  the  system  manager.   For  example,  when  a 
personnel  laboratory  provides  the  personnel  subsystem  for  a  system,  there 
is  no  way  to  evaluate  precisely  the  contribution  of  the  personnel  sub- 
system to  the  total  system  effectiveness.   Item  5  is  also  closely  related 
to  implementation,  and  has  a  strong  judgmental  component.   Accordingly, 
all  but  the  last  mentioned  criterion  reflect  the  context  in  which  the 
other  criteria  are  often  used — namely,  in  advisory  panel  reviews.   Peer 
ratings  will  be  discussed  below  under  methods. 

Additional  criteria  which  have  been  used  in  appraising  and  evalu- 
ating R&D  can  also  be  compiled  from  other  sources  (Hall  &  Lawler,  1971; 
Hoshovsky,  1970;  Pelz  &  Andrews,  1966;  US  Atomic  Agency  Manual,  Appendix) 
A  partial  listing  includes  the  following,  which  are  shown  irrespective  of 
the  distinction  between  performance  measures  vs.  quality  measures: 


Inventiveness  in  advancing  technologies 

Quality  and  originality  of  ideas  and  proposals 

Discernment  in  determining  when  lines  of  inquiry  become 
unprofitable 

Stature  of  individuals  and  organization  within  the  scien- 
tific community 

Overall  manpower  levels  in  relation  to  work  output 

Effectiveness  of  personnel  policies  in  attracting  and  re- 
taining qualified  technical  staff 

Net  change  in  R&D  budget  during  the  last  year 

Number  of  new  internally  funded  projects 

Percent  of  projects  meeting  schedule 

Number  of  projects/contracts  renewed 

Percent  of  projects  meeting  cost  budget 

Ratio  of  contracts  won  to  contracts  proposed 

Number  of  papers  in  professional  journals 

Number  of  patents  and  patent  applications 
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Number  of  technical  reports 

Frequency  of  visits  to  laboratory 

Quick  fixes,  problem  solution  events 

Citations  in  the  literature 

Requests  for  laboratory  reports,  services 

Percent  of  project  completed  to  those  started 


D.   METHODS 

Two  methods  of  evaluation  have  been  mentioned  in  the  discussion. 
One  was  the  use  of  advisory  panels,  and  the  other  was  peer  ratings.   Since 
advisory  panels  use  ad  hoc  procedures  which  depend  on  their  purpose  and 
constituency,  as  previously  discussed,  they  will  not  be  further  considered 
for  methodology.   Accordingly,  this  section  will  concentrate  on  the  peer 
rating  methods.   Before  entering  into  an  examination  of  peer  ratings,  how- 
ever, two  other  approaches  suggested  by  the  criteria  involved  should  be 
mentioned. 

The  first  is  system  effectiveness  as  an  indicator  of  the  effective- 
ness of  the  laboratory  which  held  primary  management  responsibilities  for 
the  system.   It  was  said  that  personnel  laboratories  provide  input  only  in 
the  form  of  subsystems  which  cannot  be  evaluated  in  any  precise  manner  for 
their  contribution  to  total  system  effectiveness.   This  is  true  in  the 
case  of  military  hardware  systems.   On  the  other  hand,  the  personnel  labor- 
atories have  prime  responsibility  for  certain  soft  systems,  such  as  computer 
models  for  personnel  planning,  personnel  selection  devices  and  procedures, 
training  packages  and  computer-aided  course  materials.   Traditional  measures 
of  worth  could  be  applied  in  such  instances — measures  such  as  reduction  in 
reaction  time,  workload,  and  costs  and  an  increase  in  efficiency.   But  in 
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this  initial  examination  of  the  laboratory  evaluation  problem,  this  area 
will  not  be  further  elaborated.   Two  reasons  might  be  given  for  this  de- 
cision.  In  the  first  place,  such  programs  are  less  than  a  majority  of  the 
total  laboratory  program.   Second,  there  is  the  complicating  factor  of 
false  starts,  seemingly  interminable  programs,  and  junked  programs  of 
this  nature  that  must  also  be  considered  in  measuring  the  effectiveness 
of  the  laboratory  with  respect  to  its  own  system  development  projects. 
Finally,  as  an  initial  procedure,  a  peer  rating  method  could  be  devised 
to  obtain  judgmental  evaluations  of  the  value,  quality,  and  effectiveness 
of  programs  in  this  area. 

The  other  area  of  evaluation  that  might  be  applied  to  the  products 
of  personnel  research  laboratories  stems  from  the  implementation  criterion, 
The  approach  would  be  to  determine  the  degree  to  which  products  of  the 
laboratory  have  been  accepted  into  system  designs  or  into  system  design 
methodology.   This  is  also  a  complicated  area  in  respect  to  an  effective- 
ness evaluation  of  the  laboratories.   For  example,  implementation  of  a 
product  does  not  necessarily  mean  that  greater  effectiveness  was  achieved. 
The  opposite  could  be  the  case.   And  again,  the  projects  that  are  termi- 
nated prior  to  implementation  decisions  must  also  be  considered  in  the 
evaluation.   Implementation  of  a  laboratory  product  in  a  particular  appli- 
cation might  also  be  less  than  100  percent  of  the  laboratory  package,  or 
the  product  might  be  significantly  modified  by  the  user  before  and  during 
implementation.   And  finally,  lack  of  implementation  by  the  user  agency 
does  not  necessarily  mean  that  the  product  was  ineffective,  since  many 
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f actors  are  involved  in  the  nonacceptance  of  innovations  in  the  Navy 
(Mecherikoff  &  Mackie,  1970).   Again,  as  with  the  system  effectiveness 
measure,  products  which  might  be  evaluated  using  the  implementation  cri- 
terion can  also  be  assessed  judgmentally  using  peer  rating  methods  as 
the  initial  step. 

The  evaluation  of  laboratory  effectiveness  and  quality  by  peer 
ratings  in  the  DoD  stems  from  the  work  of  Pelz  and  Andrews  (1966).   Their 
objective  was  not  an  evaluation  of  laboratory  effectiveness,  however. 
Rather,  they  were  interested  in  the  organizational  and  individual,  per- 
sonal variables  that  lead  to  effectiveness  among  scientists  in  R&D  settings. 
But  obviously,  the  quality  of  individual  work  in  an  organization  is  clearly 
related  to  institutional  effectiveness.   Accordingly,  a  method  to  evaluate 
the  effectiveness  of  individuals  in  an  organization  is  essentially  a  method 
for  the  evaluation  of  the  quality  of  the  institution  itself.   In  fact,  Hall 
and  Lawler  (1971)  had  R&D  organization  managers  rate  the  quality  of  their 
individual  scientists  and  then  these  ratings  were  combined  into  a  global 
rating  of  laboratory  performance  effectiveness.   They  did  this  to  circumvent 
the  difficult  problem  of  having  the  laboratory  director  rate  himself  by 
rating  the  quality  of  his  laboratory. 

Pelz  and  Andrews  had  senior  scientists  and  managers  in  R&D  organi- 
zations rate  their  scientists  on  two  criteria.   These  were:   (1)  Their 
contribution  to  general  technical  or  scientific  knowledge  within  their 
field  (in  the  last  5  years)  and   (2)  their  overall  usefulness  in  helping 
the  organization  carry  out  its  responsibilities  (within  the  last  5  years) . 
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Now  the  difficult  problems  that  accompany  a  mass  rating  scheme  such  as 
this  result  from  the  fact  that  judges  do  not  know  all  the  ratees,  do  not 
know  them  equally  well,  and  do  not  have  equal  confidence  among  themselves 
as  to  the  fineness  of  the  discriminations  they  could  make  (number  of  rat- 
ing units)  among  the  ratees.   Usual  rating  methods,  however,  cannot  handle 
gaps  in  the  data  well  and  require  a  consistent  scale  of  measurement  among 
judges.   As  a  consequence  of  these  constrictions,  considerable  forcing  of 
the  raters  is  resorted  to.   The  raters  or  judges  are  forced  to  rate  all 
ratees  using  the  same  scale  categories;  sometimes  judges  are  even  required 
to  place  an  equal  number  of  ratees  in  each  category  (to  prevent  them  from 
all  piling  up  at  the  high  end).  Thus,  the  convenience  of  the  investigator 
is  satisfied  with  unknown,  but  perhaps  considerable,  loss  in  the  validity 
of  the  ratings.   That  is,  adding  noise  to  a  set  of  data,  even  if  it  is 
done  in  a  systematic  way,  does  not  always  clarify  the  signal. 

Pelz  and  Andrews  (1966)  and  Andrews  and  Pelz  (in  preparation)  des- 
cribe how  they  were  able  to  devise  a  program  based  on  an  earlier  paper  by 
Ford  in  the  American  Mathematical  Monthly.   The  Ford  method  allowed  a  judge 
to  eliminate  those  ratees  he  did  not  know,  create  as  many  rating  categories 
as  he  wished,  and  to  place  as  many  individuals  as  he  desired  in  any  one 
category  (but  not  more  than  one-third  of  all  individuals  being  rated) .   As 
a  result  of  the  freedom  permitted,  the  judges  were  able  to  space  their  sub- 
jects over  the  categories  they  had  created  and  could  evaluate  approximately 
50  scientists  on  the  two  criteria  in  less  than  one  hour  without  obvious 
strain.   This  flexibility  of  their  rating  procedure  was  undoubtedly  a  major 
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factor  in  the  amount  and  usefulness  of  the  data  Pelz  and  Andrews  were  able 
to  obtain. 

The  Ford  procedure  is  based  on  a  win-loss  matrix  which  shows  the 
probability  that  each  ratee  will  be  ranked  above  or  below  every  other 
ratee.   Then,  by  an  iterative  process,  weights  are  derived  for  each  ratee 
so  that,  when  ratee-ratee  comparisons  are  made  using  the  weights,  the 
original  matrix  will  be  recaptured.   Ordinal  scaling  (ranking)  of  the 
weights  then  provide  the  ranking  of  each  ratee.   The  program  is  written 
in  Fortran  for  an  IBM  360/40  system  and  will  handle  up  to  130  judges  who 
may  use  up  to  130  ranked  categories  in  evaluating  up  to  130  objects 
(Andrews  &  Pelz,  in  preparation). 

Dr.  Maurice  Apstein  of  the  Harry  Diamond  Laboratories  used  a  modi- 
fied Pelz  technique  in  which  laboratories  instead  of  individuals  were 
ranked  by  peers  (Glass,  1969).   The  peers,  or  judges,  were  professional 
technical  people  with  considerable  experience  in  R&D.   The  rankings  were 
made  to  serve  as  the  criterion  variable  against  which  to  evaluate  the  size 
of  the  contract  effort  administered  by  a  laboratory;  that  is,  interest 
was  in  determining  the  effect  that  the  size  of  the  contract  effort  admin- 
istered by  the  laboratory  had  on  the  general  overall  excellence  of  the 
laboratory. 

The  current  DoD  study  (Glass,  1969)  in  this  mold  was  initiated  by 
Mr.  Evan  D.  Anderson  of  the  Office  for  Laboratory  Management,  ODDR&E, 
before  his  transfer  to  the  Federal  Commission  on  Procurement.   The  study 
is  much  broader  in  size  and  scope  than  Apstein' s.   There  were  250  judges 
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in  the  sample  with  up  to  400  being  planned  for  the  ultimate  sample.   The 
breakdown  of  judges  was  as  follows: 


(1)  DoD  laboratory  directors 

(2)  R&D  managers  and  technical  specialists  within  the 

Federal  Government 

(3)  Industrial  technical  specialists,  consultants  and 

professionals  from  nonprofit  organizations 

(4)  Scientists  and  engineers  in  academic  institutions 

(5)  Technical  specialists  in  DoD  program  management  and 

system  project  offices 


The  larger  number  of  judges  permitted  their  assignment  into  the  foregoing 
categories  for  analysis  of  rankings.   Accordingly,  judgments  of  laboratories 
could  be  made  on  rather  specific  performance  factors  in  contrast  to  global 
rankings  of  "technical  competence."   This  permits  a  finer  analysis  of  the 
competence  of  the  various  laboratories  and  of  particular  interest  are  the 
rankings  of  the  customer's  of  the  laboratories. 

As  with  the  Pelz  and  Andrews  study  and  the  Apstein  study,  the  ob- 
jective of  the  current  DoD  study  is  not  simply  a  relative  ranking  of  the 
laboratories.   Here  again,  the  rankings — now  available  on  different  ranking 
criteria--provide  the  basis,  the  dependent  variable,  the  criterion  against 
which  many  factors  relating  to  R&D  management  will  be  analyzed  with  the 
ultimate  purpose  of  achieving  a  better  understanding  of  the  R&D  process  in 
order  to  administer  it  better. 
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IV.   INNOVATION  IN  RESEARCH  AND  DEVELOPMENT  ENVIRONMENTS 

There  is  little  in  the  scientific  literature  on  innovation  in 
R&D  laboratories.   Perhaps  the  reason  for  this  is  that  scientists  and 
technical  personnel  are  by  the  very  nature  of  their  professions  assumed 
to  be  innovative.   Why  do  research  on  innovation  among  innovators? 
According  to  Argyris  (1968,  p.  354), 


There  is,  in  short,  almost  nothing  that  can  be  said  about 
redesigning  and  changing  research  and  development  organi- 
zations that  is  based  upon  empirical  research  and  experi- 
mentation within  these  types  of  organizations.   It  is 
ironic  but  regrettably  true  that  the  very  organizations 
charged  with  research  and  innovation  have  been  responsible 
for  generating  little  research  and  less  innovation  about 
their  own  effectiveness. 


Because  of  this  situation,  factors  that  might  be  important  for  innovation 
within  laboratories  will  be  derived  from  a  consideration  of  implementation 
problems  in  reverse.   That  is,  problems  that  arise  in  the  implementation 
of  R&D  products  in  user  organizations  often  involve  innovations  in  tech- 
niques, procedures,  and/or  materiel.   Innovations  within  a  R&D  laboratory, 
then,  might  be  considered  a  special  case  of  implementation  in  which  the 
innovation  is  developed  elsewhere  and  brought  into  the  laboratory  (even 
though  the  innovation  might  have  been  developed  by  the  laboratory,  itself). 

The  problems  of  implementation  associated  with  innovative  items 
can  be  studied  from  several  approaches  and  to  varying  depths  of  detail. 
Mecherikoff  and  Mackie  (1970)  analyzed  the  problem  of  acceptance  of  inno- 
vations in  the  Navy  as  a  problem  in  attitude  change.   A  review  of  the  ex- 
tensive literature  on  attitude  change,  a  major  area  of  personality  and 
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social  psychology,  did  not  provide  them  with  concrete  answers  or  handy 
tools  for  overcoming  negative  attitudes  toward  innovations.   While  their 
study  is  couched  in  the  technical  terminology  of  attitude  change,  their 
structuring  of  the  problem  and  recommendations — based  on  case  studies  as 
well  as  the  literature  review — is  essentially  a  tract  on  salesmanship. 
So  while  their  recommendations — actually  guidance — regarding  appropriate 
procedures  seem  to  be  fundamentally  sound,  they,  like  principles  of  sales- 
manship, are  probably  only  as  good  as  the  skill  of  the  salesman  (the  change 
advocate)  in  any  particular  situation. 

The  factors  involved  in  attitude  change  regarding  an  innovation 
are  listed  by  them  as  involving  at  least  the  following:   (1)  The  person 
advocating  the  change,   (2)  the  communication  or  message  that  accompanies 
the  effort,   (3)  the  group  factors  which  affect  an  individual's  propensity 
for  change,   (4)  individual  characteristics  that  result  in  different  re- 
sponses to  innovative  efforts,  and   (5)  situational  factors.   Any  one  of 
these  factors  can  take  on  many  values,  and  there  are  complex  interactions 
among  them.   Accordingly,  any  specific  occurrence  of  an  innovation  problem 
may  have  an  infinite  number  of  alternative  courses  that  the  attitude  change 
process  might  take.   Because  of  the  impossibility  of  placing  probabilities 
on  these  alternative  courses  with  any  degree  of  confidence,  an  approach 
that  attempts  to  break  down  the  innovation  problem  into  such  fragmented 
detail  does  not  have  practical  applications.   The  approach,  of  course,  may 
have  considerable  heuristic  value. 

Those  involved  in  the  Northwestern  University  Program  of  Research 
on  the  Management  of  Research  and  Development  have  taken  a  more  molar 
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approach  to  the  implementation  of  R&D  products  (Radnor,  Rubenstein,  & 
Tansik,  1970).   Considering  the  implementation  problem  as  a  special  case 
of  the  general  phase-transition  process  in  R&D,  they  reach  essentially 
the  same  impasse  as  Mecherikoff  and  Mackie.   They  state  that  "Many  new 
products,  product  lines,  and  technologies  fail  initially,  and  sometimes 
remain  failures,  because  of  the  inability  of  the  organization  and  its 
members  to  adapt  their  attitudes  and  behavior  in  a  manner  required  by 
the  new  venture"  (Radnor,  Rubenstein,  &  Tansik,  1970,  p.  969).   They  list 
the  following  factors  that  underly  implementation  problems  in  the  R&D 
environment  (Radnor,  et.  al . ,  1970,  p.  971): 


1.  Recognition  of  the  need  for  an  item. 

2.  Willingness  of  the  individuals  in  the  receiving  unit 
to  interrupt  ongoing  work  to  handle  something  new. 

3.  Technical  mismatch  in  understanding  of  the  specifica- 
tions of  the  item. 

4.  Mismatch  in  understanding  of  objectives  of  the  project 
or  task. 

5.  Pre-existing  relations  of  trust  or  confidence  between 
the  parties  to  an  implementation  transaction. 

6.  Degree  of  involvement  in  stages  of  a  project. 

7.  Self-interest. 

8.  Urgency. 

9.  Perceived  threat. 

10.  Level  of  managerial  support. 

11.  Point  in  time  at  which  a  management  committment  is 
made  to  the  project — i.e.,  the  decision  to  set  up  a 
formal  project  mechanism. 


Their  model  of  the  phase-transition  process  that  is  implementation  is 
more  specific  and  worked  out  in  fuller  detail  compared  with  the  large 
"black  box"  diagrams  of  Mecherikoff  and  Mackie.   Nevertheless,  it  is  still 
a  theoretical  model  of  the  process  and  not  a  practical,  workable  tool. 
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Following  their  critical  review  of  the  implementation  problem, 
they  list  some  methods  that  have  been  reportedly  successful  in  implemen- 
tation, but  warn  that  they  have  been  used  in  projects  that  have  failed, 
too.   The  methods  are  (Radnor,  et.  al .  ,  1970,  p.  989): 


1.  Assuring  that  there  is  a  clear  and  recognized  need 
for  the  results  at  the  time  the  project  is  undertaken. 

2.  Involvement  of  the  ultimate  user  of  the  results  early 
in  the  process,  and  maintaining  communication  through- 
out the  project. 

3.  Focus  of  the  direction  or  strategy  for  the  project  in 
an  individual  or  a  small  group  that  can  review  and  make 
decisions  about  changes  in  direction  or  level  of  effort, 

4.  Having  top  management  support  and  enthusiasm. 

5.  Allowing  or  encouraging  researchers  to  follow  projects 
into  application  and  make  careers  there,  if  they  so 
desire. 


They  conclude,  from  the  wide  diversity  of  methods  used  and  the  lack  of 
consistent  cause-effect  relations  between  methods  used  and  project  success, 
that  much  more  research  is  needed  in  the  area  before  there  will  be  a  rea- 
sonable theory  of  implementation,  let  alone  a  handbook  or  set  of  algorithms 
for  carrying  out  implementation.   Carried  over  into  the  implementation  of 
innovations  within  an  R&D  laboratory,  an  even  stronger  conclusion  can  be 
made:   there  are  no  administrative  and  managerial  practices  that  are  guar- 
anteed to  succeed. 

What,  then,  could  management  do  to  bring  innovations  into  the  lab- 
oratory with  at  least  some  modicum  of  success?   In  the  first  place,  the 
guidance  that  can  be  gained  from  a  study  and  application  of  principles 
developed  in  documents,  such  as  those  cited  in  the  discussion  above,  should 
not  be  ignored.   But  there  would  seem  to  be  a  lesson  that  can  be  obtained 
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from  the  two  lists  provided  by  Radnor  et_.  al.  that  is  especially  appro- 
priate for  R&D  management.   If  we  remember  that  the  implementation  prob- 
lem as  discussed  arises  from  the  user  environment,  the  picture  may  not 
be  so  bleak  in  the  R&D  environment.   That  is,  implementation  in  reverse 
is  not  exactly  the  same  as  implementation  in  the  forward  direction  because 
innovation  in  the  R&D  environment  is  the  accepted  way  of  life  and  the  path 
to  success.   It  is  administered  out  of  the  environment  by  the  dysfunctional 
control  processes  so  aptly  described  by  Argyris.   Innovation  in  the  labora- 
tory should,  then,  follow  as  a  direct  result  of  good  administration  of  the 
laboratory  that  creates  in  its  individual  members  a  need  for  innovation, 
the  willingness  to  give  time  and  effort  to  it,  an  involvement  in  innovative 
projects,  a  feeling  of  a  sense  of  urgency  for  the  development  and  use  of 
innovative  approaches,  and  an  opportunity  to  cash  in  on  successful  appli- 
cation of  innovative  techniques  and  procedures.   These  factors,  taken  from 
the  Radnor,  e_t.  a_l.  lists,  are  but  a  specific  example  of  the  characteristics 
of  positive  laboratory  pressures  described  earlier  from  the  works  of  Hall 
and  Lawler:   quality,  time,  and  financial  responsibility  pressures  and  a 
deep  sense  of  involvement  in  the  total  life  framework  of  the  laboratory. 
As  shown  by  Merchikoff  and  Mackie,  the  pressure  points  where  action  could 
be  taken  to  accomplish  these  ends  are  the  individual  participant  and  the 
group  and  situational  factors.   Referred  to  the  laboratory  environment, 
these  are  precisely  the  dimensions  of  the  scientist-laboratory  interface 
shown  in  figure  2  and  described  in  section  II  c,  above.   The  problem  for 
the  laboratory  manager  is  to  determine  which  of • the  many  interfaces,  when 
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properly  manipulated,  will  provide  the  most  payoff  in  the  direction  de- 
sired.  An  approach  that  may  solve  this  problem  satisfactorily  in  the 
Navy's  personnel  research  laboratories  is  given  in  section  V  B,  recommen- 
dations for  innovation. 
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V.   RECOMMENDATIONS 
A.   RECOMMENDATIONS  FOR  EFFECTIVENESS  EVALUATION 

It  is  known  that  the  usual  management  procedures  for  ensuring  ade- 
quate performance  of  a  laboratory,  such  as  coordinating  conferences,  reports, 
person-to-person  contacts,  and  so  forth,  are  fully  used  in  the  management 
of  the  Navy's  personnel  research  laboratories.   Further  elaboration  of  this 
area  as  a  means  for  effectiveness  evaluation  is  not  contemplated. 

From  the  preceding  discussion  involving  the  use  of  advisory  panels 
to  evaluate  the  quality  and  effectiveness  of  a  laboratory,  it  is  recommended 
that  such  a  panel  or  panels  be  established.    The  purpose  of  such  a  panel 
should  be  to  assess  the  health  of  current  programs;  the  quality  of  personnel 
and  the  techniques  being  employed;  the  relevance  and  meaningf ulness  of  the 
total  program  configuration  of  the  laboratories;  and  methods  being  employed 
to  upgrade  facilities,  skills,  and  capabilities.   The  panel  should  include 
technical  personnel  and  representatives  of  consumer  organizations,  as  in 
the  case  of  the  panel  for  the  current  DoD  peer  rating  effort  in  view  of  the 
broad  spectrum  of  projects  worked  on  in  the  laboratories.   The  reporting 
level  will  probably  be  at  the  Bureau  of  Naval  Personnel  level,  but  the  ac- 
tivities of  the  panel  should  emphasize  interactions  with  personnel  at  the 
laboratory  level  to  ensure  a  feeling  of  participatory  management  on  the  part 
of  the  laboratory  people.   It  would  seem  advisable  to  have  a  subpanel  or 
another  panel  at  the  laboratory  director's  level  that  is  composed  strictly 
of  technical  personnel  to  provide  advice  and  guidance  to  the  laboratory  as 
requested. 

1.   While  this' study  was  in  progress,  a  Laboratory  Advisory  Board  for 
Personnel  Laboratories  has  been  established  under  the  aegis  of  the  Naval 
Research  Advisory  Committee  (NRAC) .   The  primary  purpose  of  the  advisory 
board  is  to  provide  NRAC  background  information  for  its  advice  on  Navy  RDT&E 
matters  to  the  Assistant  Secretary  of  Navy  (R&D).   Its  specific  areas  of 
evaluation  are  similar  to  those  discussed  here.   The  members  are  all  tech- 
nical personnel  under  the  chairmanship  of  Dr.  Joseph  W.  Rigney ,  Department 
of  Psychology,  University  of  Southern  California. 
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In  the  case  of  individual  laboratories,  some  estimate  of  the  quality 
or  effectiveness  of  its  products — rather  than  the  laboratory  as  an  institu- 
tion— is  desired.   Currently,  the  formal  reports  of  the  laboratories  are 
accompanied  by  an  "Evaluation  of  Report"  form  which  requests  the  reader  to 
evaluate  the  report  and  mail  in  the  evaluation  form.   A  similar  form  exists 
for  end-product  evaluations  that  are  not  in  the  formal  report  series.   The 
report  evaluation  asks  the  reviewer  to  rate  the  report  as  low,  average,  or 
high  on  several  categories  which  are  related  to  usefulness,  timeliness, 
accuracy,  and  format  factors.   In  addition,  open-ended  questions  and  room 
for  comments  are  provided.   The  questions  also  emphasize  an  evaluation  of 
the  usefulness  of  the  report  and  the  usefulness  of  the  laboratory's  products. 
Unfortunately,  the  return  rate  of  the  forms  is  essentially  nil.   A  similar 
form  was  in  use  by  the  US  Air  Forces  Human  Resources  Laboratory,  with  the 
same  result.   The  latter  has  been  discontinued. 

It  is  suggested  that  the  Pelz-Andrews  technique  be  modified  and 
applied  to  the  laboratory's  products  as  the  rated  items  in  lieu  of  individ- 
ual professionals  or  individual  laboratories.   The  products  could  be  rated 
at  the  program  (task  area)  level  or  at  the  level  of  formal  reports  that  are 
terminal,  comprehensive  reports  of  a  project.   Probably  the  latter  would 
present  a  problem  in  selection  or  a  problem  in  too  many  products  for  evalua- 
tion.  The  decision  would  have  to  be  worked  out  with  the  personnel  R&D 
management  structure.   It  is  further  suggested  that  the  panels  be  of  two 
distinct  types.   A  type-one  panel  would  be  composed  of  technical  personnel 
from  the  user  agencies  who  could  be  expected  to  be  familiar  with  the  products 
of  the  laboratories.   This  would  provide  the  all-important  consumer  appraisal 
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of  a  laboratory's  products.   A  type-two  panel  would  be  composed  of  only 
technical/professional  personnel  from  the  laboratory.   This  would  provide 
the  important  factor  of  laboratory  participation  in  the  evaluation  process,  and 
indirectly  a  voice  in  the  laboratory's  R&D  program.   The  technical  panel 
could  be  supplemented  by  personnel  from  outside  the  laboratory  who  are  fa- 
miliar with  its  products  to  provide  another  facet  to  the  evaluation.   All 
of  the  panels  would  not  need  to  be  implemented.   The  consumer  panel  would 
probably  provide  management  with  the  most  useful  information  with  respect 
to  effectiveness.   The  in-house,  type-two  panel  would  be  the  easiest  to 
implement . 

The  criteria  for  the  ratings  would  be  different  for  the  two  panels. 
The  type-one  panel  would  use  two  criteria  which  could  be  called   (1)  use- 
fulness and    (2)  influence.   The  usefulness  criterion  would  include  the 
area  of  evaluation  portrayed  on  the  Evaluation  of  Report  form  and  would 
emphasize  short-term,  direct  applications  of  the  product.   Included  in  the 
criterion  would  be  factors  such  as  usefulness,  timeliness,  completeness, 
accuracy,  adequacy  of  data,  adequacy  of  approach,  convenience  of  format. 
The  specific  wording  of  the  criterion  would  be  worked  out  with  personnel 
research,  management  personnel.   The  second  criterion,  dealing  with  the 
influence  of  the  report,  will  emphasize  its  long-term  usefulness  according 
to  its  potential  influence  on  concepts,  planning,  and  decision-making  in 
the  user  agencies. 

The  criteria  for  the  type-two  panel  would  also  number  two:   (1)  tech- 
nical-scientific value  and   (2)  mission  appropriateness.   The  first  criter- 
ion would  be  similar  to  that  used  by  Pelz  and  Andrews  regarding  the  work  of 
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individual  scientists.   The  product  would  be  rated  for  its  value  in  con- 
tributing to  knowledge  in  the  field  and  especially  its  value  in  furthering 
the  work  of  the  laboratory.   The  second  criterion  would  permit  the  labora- 
tory personnel  to  express  their  thoughts  on  what  the  laboratory  should  be 
doing,  considering  its  general  mission.   In  other  words,  the  panel  would 
rate  its  programs  as  most  appropriate  or  representative  of  what  the  mission 
calls  for.   The  mission  statements,  as  of  this  writing,  are  those  in  DDR&E 
Management  Analysis  Report,  MAR  70-4  (pages  87  and  95). 

The  method  of  analysis  would  involve  the  use  of  the  University  of 
Michigan  program,  available  from  the  Survey  Research  Center,  for  the  Ford 
analysis.   The  derived  weights  for  the  products  analyzed  could  serve  as 
the  dependent  variable  for  further  analyses  based  on  characteristics  of 
the  raters  and  of  the  rated  items.   Such  analyses  would  provide  a  fuller 
understanding  of  what  factors  constitute  the  effective  and  noneffective 
products  or  programs  of  the  laboratories.   They  would  also  provide  input 
for  managerial  and  administrative  practices  to  enhance  the  implementation 
of  innovative  techniques  and  procedures. 

A  difficult  problem  in  carrying  out  the  recommended  ratings  would 
be  an  appropriate  definition  of  the  "consumer"  for  a  type  I  panel.   Because 
of  the  diversification  of  the  areas  of  specialization  of  the  Navy's  per- 
sonnel research  laboratories,  potential  customers  will  not  be  a  homogenous 
group.   Accordingly,  rankings  obtained  through  a  type  I  panel  will  reflect 
the  constitution  of  the  panel.   In  addition,  there  may  be  an  insufficient 
commonality  of  items  rated  to  provide  a  reliable  ranking  over  a  majority 
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of  the  items.   In  the  extreme  case,  each  of  a  few  judges  may  rate  completely 
different  items,  for  example.   Probably  the  best  way  to  overcome  these  po- 
tential problem  areas  would  be  to  use  many  raters  (as  in  the  DoD  laboratory 
study  mentioned  previously).   But  here,  practical  considerations  and  the 
limited  number  of  potential  raters  may  present  other  problems.   In  the  final 
analysis,  however,  the  rankings  by  a  type  I  panel  constitute  a  managerial 
tool  for  providing  needed  information,  and  any  bias  in  the  constitution  of 
the  rater  sample  should  reflect  the  desires  and  purposes  of  management  in 
conducting  the  ratings. 

The  procedure  of  evaluating  laboratory  products  by  peer  ratings 
could  be  done  at  periodic  intervals,  say  approximately  three  years,  to  main- 
tain a  feel  for  the  quality  and  effectiveness  of  the  laboratory's  work. 

As  long-range  programs,  it  is  recommended  that  studies  be  initiated 
to  determine  how  to  assess  the  contribution  of  personnel  research  laboratory 
products  to  total  system  effectiveness  and  how  to  use  implementation  as  a 
criterion  to  evaluate  effectiveness  of  the  laboratory's  output. 
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B.   RECOMMENDATIONS  FOR  INNOVATION  IMPLEMENTATION 

Interviews  at  the  Navy's  personnel  research  laboratories  indicate 
that  many  specific  attempts  to  upgrade  the  professional  excellence  of  the 
laboratory  are  being  made.   They  include  the  procurement  of  outside  speakers, 
tutorials  on  specific  techniques,  attendance  at  professional  meetings, 
bringing  in  specially  qualified  reserve  officers  for  their  "summer  tours," 
sponsoring  symposia,  attendance  at  short  courses,  and  the  other  commonly 
used  procedures.   In  one  instance,  for  example,  the  developers  of  the  auto- 
mated operational  sequence  diagramming  technique  at  San  Diego  went  to  the 
Washington  laboratory  to  put  on  a  tutorial  on  the  method.   The  effects  of 
these  attempts  is  not  known  but  assumed  to  be  beneficial.   (The  usual  case.) 

It  was  concluded  in  the  section  on  innovation  (section  IV  D)  that 
innovation  in  the  laboratory  is  a  problem  in  manipulating  the  appropriate 
variables  in  the  scientist-laboratory  interface  to  bring  about  the  climate 
that  would,  of  itself,  create  a  motivation  for  innovation.   Then,  any 
specific  innovative  attempt  or  project  would  have  a  greater  chance  of  succeed- 
ing.  The  problem  lies  in  deciding  which  set  of  variables  to  manipulate. 

In  order  to  determine  what  set  of  variables  to  manipulate,  one 
must  know  the  important  interactions  and  their  outcomes  of  the  input  and 
process  variables  shown  in  figure  2.   A  comprehensive  study  showing  impor- 
tant relationships  of  this  type  is  the  Pelz-Andrews  monograph  previously 
referred  to  on  many  occasions.   A  simple  way  to  answer  the  question  of  what 
to  do  lies  in  obtaining  measurements  on  the  important  variables  of  the  Pelz- 
Andrews  study  for  a  particular  laboratory  and  then  going  to  the  study  to 
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determine  what  manipulations  will  produce  the  desired  climate.   For  example, 
if  one  group  of  PhD  scientists  turned  up  in  a  predominantly  research-ori- 
ented work  unit  in  which  the  decision  controls  for  any  one  person  were  very 
few,  he  would  find  from  the  Pelz-Andrews  monograph  that  having  four  decision- 
controls  (decreasing  freedom!)  resulted  in  a  higher  volume  and  quality  of 
output  for  such  scientists.   The  indicated  action  is  obvious. 

Such  an  approach  could  be  taken  at  several  levels  of  complexity  with 
respect  to  data  collection.   A  level-one  analysis  (simple  date  collection) 
would  use  the  following  variables: 

Individual  variables:   Age;  education;  time  since  BA/S,  MA/S, 
PhD;  and  area  of  specialization. 

Work  unit  variables:   Identification  of  unit,  size,  predomi- 
nant orientation. 

The  combination  of  the  individual  variables  with  the  work-unit  variables 

will  lead  to  many  practical  management  procedures  for  moving  the  climate 

toward  greater  effectiveness  of  the  work  unit,  including  implementation  of 

innovative  methods.   The  value  for  these  variables  could  be  obtained  from 

individual  records,  organizational  charts,  and  unit  charters.   In  addition, 

management  could  provide  estimates  as  to  some  of  the  control  variables 

that  might  apply  to  the  work  units,  thus  enrichening  the  analysis.   If  the 

laboratory  products  evaluated  in  the  peer  ratings  can  be  traced  by  source 

to  the  work  units,  the  relationship  of  work-unit  characteristics  to  quality 

of  output  could  be  analyzed  using  a  multivariate  procedure  described  below. 

A  more  complex  level  of  analysis  would  involve  the  creation  of 

appropriate  questionnaires  to  obtain  data  from  individual  scientists  in  the 
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laboratories .   Of  particular  interest  would  be  the  obtaining  of  additional 
dependent  measures  for  multivariate  analyses.   These  would  be  the  number 
of  papers  published  and  presentations  made  in  a  specified  period  of  time 
and  the  number  of  laboratory  reports  authored/coauthored  in  that  period. 
Again,  it  would  be  desirable  to  identify  the  work  unit  and  the  individual 
with  specific  products  or  program  areas  that  have  been  peer  rated.   Multi- 
variate analyses  of  the  input  and  process  variables  using  the  peer  ratings 
and  individual  production  as  the  dependent  variables  would  provide  a  rich 
picture  of  the  factors  within  the  laboratory  associated  with  effective  and 
noneffective  individuals  and  work  units.   A  multivariate  technique,  Multiple 
Classification  Analysis,  has  been  obtained  from  the  Survey  Research  Center 
that  is  particularly  adaptable  to  this  analysis  (Andrews,  Morgan  &  Sonquist, 
1969).   This  technique  and  the  computer  program  listed  were  used  for  the 
analyses  in  the  Pelz  and  Andrews  study  of  scientists  in  organizations. 
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